ek iar purse ee. a a a S 


DIGITAL DISRUPTION IN 
TEACHING AND TESTING 


ASSESSMENTS, BIG DATA, AND THE 
TRANSFORMATION OF SCHOOLING 


Edited by 
CLAIRE WYATT-SMITH, BOB LINGARD, 
AND ELIZABETH HECK 


3940J11N0Y 


DIGITAL DISRUPTION IN TEACHING 
AND TESTING 


This book provides a significant contribution to the increasing conversation 
concerning the place of big data in education. Offering a multidisciplinary 
approach with a diversity of perspectives from international scholars and industry 
experts, chapter authors engage in both research- and industry-informed discussions 
and analyses on the place of big data in education, particularly as it pertains to large- 
scale and ongoing assessment practices moving into the digital space. This volume 
offers an innovative, practical, and international view of the future of current 
opportunities and challenges in education and the place of assessment in this context. 


Claire Wyatt-Smith is Professor of Assessment and Literacy. She is the Director 
of the Institute for Learning Sciences & Teacher Education at the Australian 
Catholic University. 


Bob Lingard is a Professorial Fellow at the Institute for Learning Sciences & 
Teacher Education at the Australian Catholic University and Emeritus Professor 
of Education at the University of Queensland. 


Elizabeth Heck is a Research Assistant at the Institute for Learning Sciences & 
Teacher Education at the Australian Catholic University. 


CRITICAL SOCIAL THOUGHT 


The Critical Turn in Education 
From Marxist Critique to Poststructuralist Feminism to Critical Theories of Race 
Isaac Gottesman 


The Struggle for the Soul of Teacher Education 
Kenneth M. Zeichner 


College Curriculum and the Crossroads 
Women of Color Reflect and Resist 
Ed. By Kirsten T. Edwards and Maria del Guadalupe Davidson 


Educating for Critical Consciousness 
Ed. By George Yancy 


Hidden Markets 
Public Policy and the Push to Privatize Education 
Patricia Burch 


Digital Disruption in Teaching and Testing 
Assessments, Big Data, and the Transformation of Schooling 
Ed. By Claire Wyatt-Smith, Bob Lingard, and Elizabeth Heck 


For more information about this series, please visit: https://www.routledge.com/ 
Critical-Social-Thought/book-series/SE0807 


DIGITAL DISRUPTION 
IN TEACHING AND 
TESTING 

ASSESSMENTS, BIG DATA, AND 


THE TRANSFORMATION OF 
SCHOOLING 


Edited by Claire Wyatt-Smith, Bob Lingard, and 
Elizabeth Heck 


Z} Routledge 
g Taylor & Francis Group 


NEW YORK AND LONDON 


First published 2021 
by Routledge 
52 Vanderbilt Avenue, New York, NY 10017 


and by Routledge 
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN 


Routledge is an imprint of the Taylor & Francis Group, an informa business 
© 2021 Taylor & Francis 


The right of Claire Wyatt-Smith, Bob Lingard, Elizabeth Heck to be identified 
as the authors of the editorial material, and of the authors for their individual 
chapters, has been asserted in accordance with sections 77 and 78 of the 
Copyright, Designs and Patents Act 1988. 


The Open Access version of this book, available at www.taylorfrancis.com, 
has been made available under a Creative Commons Attribution-Non 
Commercial-No Derivatives 4.0 license. 


Trademark notice: Product or corporate names may be trademarks or registered 
trademarks, and are used only for identification and explanation without intent 
to infringe. 


Library of Congress Cataloging-in-Publication Data 

Names: Wyatt-Smith, Claire, editor. | Lingard, Bob, editor. | Heck, 
Elizabeth, editor. 

Title: Digital disruption in teaching and testing : assessments, big data, and the 
transformation of schooling / edited by Claire Wyatt-Smith, Bob Lingard, 
Elizabeth Heck. 

Description: New York, NY : Routledge, 2021. | Includes bibliographical 
references and index. 

Identifiers: LCCN 2020049453 (print) | LCCN 2020049454 (ebook) | 
ISBN 9780367493356 (hardback) | ISBN 9780367493325 (paperback) | 
ISBN 9781003045793 (ebook) 

Subjects: LCSH: Education—Data processing. | Big data. 

Classification: LCC LB1028.43 .D54 2021 (print) | LCC LB1028.43 (ebook) | 
DDC 371.33/4-dc23 

LC record available at https://lccn.loc.gov/2020049453 

LC ebook record available at https://lccn.loc.gov/2020049454 


ISBN: 978-0-367-49335-6 (hbk) 
ISBN: 978-0-367-49332-5 (pbk) 
ISBN: 978-1-003-04579-3 (ebk) 


DOI: 10.4324/9781003045793 


Typeset in Bembo 
by Taylor & Francis Books 


CONTENTS 


List of Figures 

List of Tables 
Foreword 
Acknowledgements 
Editor Biographies 
List of Contributors 
List of Abbreviations 


1 Transforming Schooling through Digital Disruption: Big Data, 
Policy, Teaching, and Assessment 
Bob Lingard, Claire Wyatt-Smith and Elizabeth Heck 


Introduction 1 
Backdrop and Contexts 3 
Policy and Assessment Frames 9 
Education Policy 9 
Working Together — Policy and Assessment Frames in the 
Digital 13 
Digital Disruption in and through Assessment and Testing 13 
Digital Disruption in and through Big Data 19 
Digital Disruption in Teaching 23 
Digital Literacy and Data Literacy for Teachers 24 
Chapters in the Collection 26 
Conclusion 33 


vi 


Contents 


2 Automated Knowledge Discovery: Tracing the Frontiers, 


Infrastructures, and Practices of Education and Data Science 45 
Ben Williamson 


Introduction 45 

Data Science Frontiers 46 
Education Data Science 49 
Precision Learning Engineering 51 
Inscribed Bodies 55 

Conclusion 56 


Artificial Intelligence and Machine Learning: A Practical and 
Ethical Guide for Teachers 60 
Erica Southgate 


Introduction 60 
What Is AI? 61 
What Is Machine Learning (ML)? 63 
How Is AI Used in Education? 65 
What Are the Ethical Concerns with AI and Some of the Governance 
Implications? 67 
1. Awareness 68 
2. Explainability 68 
3. Fairness 69 
4. Transparency 69 
5. Accountability 70 
Conclusion 70 


The Relationship between Humans and Machines in Public 
Policy 75 
Brian Lee-Archer 


Introduction 75 

Digital: It Is Not New, and It Is All around Us 76 

Problem Solving through AI 80 

Building Confidence to Address the Risks 82 

Action and Reaction 84 

Innovation, Regulation, and Oversight 85 

Conclusion: Better Public Policy through Digital 86 
1. Create the Environment for Innovation 86 
2. Nurture an Ethical Use Culture 87 


Contents vii 


3. Make Risk Your Friend 87 
4. Success in the Digital Realm Relies on Structure 87 


5 Amazon Go for Education? Artificial Intelligence, Disruption, 
and Intensification 90 
Kalervo N. Gulson, Andrew Murphie, and Kevin Witzenberger 


Introduction 90 
Conceptual and Methodological Note: Speculation as beyond Gray 
Literature 92 
Intensification 93 
Datafication and Data Infrastructures 93 
Pre-automation and the Structuring of Professional 
Obsolescence 95 
Disruption 96 
Human-machine Learning 96 
Education Technology Platforms as Education Providers 97 
Literacies and Skills in a “Third Media Revolution” 99 
Avoiding Technological Somnambulism: Enlarging Our 
Considerations of AI in Education 101 


6 Pearson’s Digital Transformation and the Disruption of Public 
Education 107 
Anna Hogan and Sam Sellar 


Introduction 107 

The Digital Transformation of the GEI 108 

Methods 110 
Charting the Promotion of Digital Learning in Pearson’s 

“Open Ideas” Forum 111 

Digital Transformation and the Disruption of Teaching, Curriculum, 
and Assessment 115 

Disrupting Teaching 116 

Disrupting Curriculum and Instruction 117 

Disrupting Schools 119 

Conclusion 120 


7 Costs of Big Data: Challenges and Possibilities of Cost-benefit 
Analysis of ILSAs 124 
Laura C. Engel and David Rutkowski 


Introduction 124 


viii Contents 


US Participation in ILSAs: What Does It Cost? 126 
ILSA Costs in the US 128 

Possibilities and Challenges of Exploring Costs and Benefits of 
ILSAs 129 

Conclusion 133 


8 Data Infrastructures and the (Ambivalent) Effects of Rusing 
Data Interoperability: Insights from Germany 136 
Sigrid Hartong, Annina Férschler and Vito Dabisch 


Introduction 136 
The German “Context” of Datafying and Digitalizing 
Education 139 
Understanding the “Disruptive” Potential of Data Infrastructures: 
Three Examples from Germany 141 
The Transforming Infrastructures of Standardized 
Assessments 141 
The Transforming Infrastructures of State School 
Monitoring 143 
The Transforming Infrastructure of School Platforms and 
Learning Management Systems 144 
Conclusion and Outlook 146 
Disclosure Statement 147 


9 Datafication and Surveillance Capitalism: The Texas Teacher 
Evaluation and Support System (T-TESS) 152 
Jessica Holloway and Steven Lewis 


Introduction 152 
The Datafication of Teachers and Teaching: Providing Fertile Ground 
for Surveillance Capitalism 154 
Teacher Accountability and the T-TESS 157 
The T-TESS Ensemble 158 
Rendering Teacher Performance as Data: SAS Analytics 
Inc. 159 
Rendering Teacher Behavior as Data: Responsive 
Learning 159 
Rendering Teacher Experience as Data: NIET 160 
Discussion and Conclusion: The Possibilities for Surveillance 
Capitalism via the T-TESS 161 


Contents ix 


10 Governing by Dashboard: Reconfiguring Education 


11 


12 


Governance in the Global South 166 
Radhika Gorur and Ben Arnold 


Introduction 166 
Dashboards in Governance 168 
The Sociotechnical Imaginary of the GEPD 170 
The Imagined “System” 171 
The Imagined Classroom 174 
The Imagined User 175 
Conclusion and Discussion: Contradictions and Conundrums 177 


Next Generation Online Assessments, Technical Democracy, 
and Responding to Digital Disruption 182 
Greg Thompson 


Introduction 182 

Education Reform in Australia 183 

Problems with NAPLAN Census Testing 186 

The Online Formative Assessment Initiative 188 

The Logistics of Engagement 189 

New Possibilities: Owning Data and Responding to the Displacement 
of the Teacher 193 

Conclusion 196 


“Lenses on COVID-19”: Provocations 200 


Provocation #1: Digital Education in the Aftermath of COVID-19: 
Critical Hopes and Concerns 
Neil Selwyn 
Introduction 200 
Point 1: COVID-19 as cover for the Corporate Reform of 
Education 201 
Point 2: Establishing Alternate Agendas and 
Counter-narratives 203 
Conclusion 204 
Note 205 
Provocation #2: Education without Borders, Rule without Limit 
Nick Couldry 
Introduction 205 
The impact of COVID-19 206 
Conclusion 208 


x Contents 


Provocation #3: The Electric “Shock” of the COVID-19 Crisis on 
Schooling 
Anna Hogan and Ben Williamson 
Introduction 208 
New Global Policy Networks 209 
Pandemic Prototyping 209 
Private Infrastructures 210 
Pandemic Profit Making 210 
Conclusion 211 
Acknowledgement 211 
Provocation #4: Teachers, the Anti-heroes? The Global Pandemic 
Crisis and the Construction of Teachers as the Problem “Other” 
Sotiria Grek 
Introduction 211 
The Changing Narrative 212 
Conclusion 213 
Note 214 
Provocation #5: The COVID-19 Pandemic Creates Opportunities 
to Repair the Infrastructure of Public Education 214 
Sam Sellar 


Glossary 220 
Index 225 


FIGURES 


1.1 
1.2 


1.3 


10.1 


10.2 


10.3 


Test tailored design with branching 

Branching structure of NAPLAN online literacy and 
numeracy tests 

The evolution of digitally-mediated ways to assess student 
progress authentically in context 

The GEPD’s conceptualization of the education system 
(World Bank, 2019a, p. 1) 

The World Development Report: “technical and political 
factors” that “divert schools, teachers, and families from a 
focus on learning” 

Teach observation tool 


14 


15 


17 


172 


173 
175 


TABLES 


3.1 


7.1 
7.2 
U3 


Types of artificial intelligence from Southgate et al. (2019), 
adapted from Hintze (2016) 

General overview of ILSAs 

Reported overall costs for PISA 2015 and TIMSS 2015 
Calculated overall costs for PISA 2015 and TIMSS 2015 


63 
127 
129 
129 


FOREWORD 


I am writing this Introduction in the midst of the COVID-19 crisis. Truly major 
changes in how formal schooling is (or is not) being carried out are visible all 
around us. Deep-seated inequalities in access and resources are distressingly evi- 
dent. Teachers are being called upon to engage with technologies for which they 
have had little preparation and experience at the same time as they struggle to 
maintain personal connections with students. Models of curriculum and pedagogy 
that depend on a substantive understanding of and responsiveness to student and 
community knowledge and realities are being lost as physical, emotional, and 
epistemological distance is maintained. At the same time, there are calls to sus- 
pend all testing of students given the realities of hybrid forms of mostly distance 
education that have been put in place. As a result of these and other conditions, a 
significant number of students are moving from public schools to private schools, 
to for-profit schools, or to homeschooling. Schools are facing a looming fiscal 
crisis that threatens to make their already difficult financial status even worse. 
Many more aspects of what has been happening in education could be listed. All 
of this makes the book you are about to read even more significant. 

There are times when I read a book that opens up new and insightful ways of 
thinking about crucial issues in education that deserve our close attention. Digital 
Disruption in Teaching and Testing: Assessments, Big Data, and the Transforma- 
tion of Schooling is definitely one of these books. I personally have weighed on 
the debates over digital technologies in education, on the over reliance on parti- 
cular kinds of evaluation and testing, and on the ways in which surveillance creates 
differential identities for students and teachers. And I have documented how such 
policies and practices often work to reproduce inequalities and impose quite 
reductive assumptions about “legitimate” knowledge, “real” learning, “good 
teaching,” “good students,” and “good schools” (Apple, 2006; Apple, 2014). 


xiv Foreword 


After reading the volume that Bob Lingard, Claire Wyatt-Smith, and Elizabeth 
Heck have produced, and the debates they provide us entry into, I now realize 
that even if the COVID-19 crisis finally passes, there is an even broader range of 
issues that are at stake when schools are increasingly deeply connected to the 
world of digital forms and practices, to the accumulation of massive amounts of 
data, and to the reliance on an ever expanding arsenal of managerial and evalua- 
tive expertise and procedures. The issues are complicated, but ignoring them 
simply increases the possibility that our often unreflective moves in this direction 
will produce unfortunate results. 

At the outset, let me emphasize that Digital Disruption in Teaching and Test- 
ing is not simply an analysis of what is wrong and what the dangers of these 
tendencies are. It is instead a very thoughtful and detailed assessment of the limits 
and possibilities of digital technologies and the sense-making processes that 
accompany them, of the rapidly growing influence of “big data” in educational 
decisions, and of changes in what counts as evidence in educational policy and 
practice. While this alone makes the book a valuable contribution, the fact that it 
is truly international in its analyses helps us better understand that context matters 
in understanding these limits and possibilities. The ambitious aims of the book 
can be seen in the following description. 

This book provides research-informed accounts of the multiple various ways 
digital disruption and data use are beginning to profoundly reshape education 
systems, policymaking practices, and the work of schools and teachers. This 
reshaping comes with implications for young people’s learning, identities, and 
experiences of schooling with significance for their futures. Such analyses...are 
necessary to begin constructive dialogues across knowledge domains and groups 
with legitimate interests in education policy and the future of schooling. These 
include students, teachers, teacher unions, parents/carers, the wider community, 
school leaders, system leaders, policymakers, and researchers...[The book opens] 
up these necessary conversations about educative and democratic uses of data and 
digital for policymaking and how these might amplify and augment the new 
teacher professionalism requiring new knowledges and expanded evaluative 
expertise. 

Given these emphases, the volume also adds to the growing tradition of critical 
analyses of the influences of neoliberalism on education and the increasing trans- 
formation of schools into sites of profit (Burch, in press). It also provides insightful 
weight to the arguments surrounding the role of schooling in what might be 
called an “epistemological war.” What is important is that which is measureable. 

In addition, the book makes a substantive contribution to the debates about 
the role both inside education and the larger society of the uses of algorithms. 
Algorithms are now used to predict and at times change tests scores on assess- 
ments that have a truly major impact on students’ futures. The results have been 
quite controversial, to say the least. Yet educators need to be much more cogni- 
zant that the uses of algorithms in society at large have been challenged by many 
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people. For example, one of the largest uses is in predicting which communities 
and which people will supposedly engage in “criminal acts.” This has all too often 
resulted in very damaging and racializing effects. As Noble states, such effects 
should make us deeply concerned about accepting “the benign instrumentality of 
technologies” such as these (Noble, 2018, p. 230; see also, O’Neil, 2016). 

These are of course extremely complicated issues. They raise ethical and political 
as well as practical questions. Do “big data” and digital resources in education actu- 
ally pay off in practice? What are their connections to neoliberal agendas? Does 
education increasingly get turned into one more site for the creation of profit? What 
are the issues of student and parent privacy? What are the effects of all of this on 
teacher autonomy and professionalism? Does it change what counts as crucial teacher 
skills and dispositions from more substantive and nuanced culturally responsive 
interpersonal practices into those that largely emphasize managerial impulses? What 
can we do to make it more likely that progressive possibilities for schools, students, 
teachers, and communities are the results? What does this mean for teacher educa- 
tion? The list of questions goes on and on. Dealing with them requires that everyone 
who can be affected by these “reforms” have a substantive voice in them. 

Digital Disruption in Teaching and Testing helps us deal honestly with these 
and other questions and provides us with significant insights into how they might 
be answered. It also provides us with a wide range of conceptual, empirical, and 
policy-related tools both to raise important questions about what we too often 
think of as “technical” solutions and to provide paths to create more thoughtful 
answers to these questions. Some of these questions and answers will require that 
we step back from our preconceptions about many of the things we too easily 
take for granted. In the process, it asks us to think deeply about what we are 
doing, why we are doing it, and what the results might actually be. 

In a number of places, I have argued that schools can be and at times are sites 
of important social transformation and of the movements that support them 
(Apple, 2013; Apple et al. 2018). This is a time when schools have become test- 
ing grounds for a number of such changes. But such transformations can often 
have contradictory results. They can be a “yes” and “no” at the same time. This 
book documents that reality; it is both a cautionary tale and a way forward. 


Michael W. Apple 
John Bascom Professor Emeritus of Curriculum and Instruction and 
Educational Policy Studies, University of Wisconsin, Madison 
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TRANSFORMING SCHOOLING 
THROUGH DIGITAL DISRUPTION 


Big Data, Policy, Teaching, and Assessment! 


Bob Lingard, Claire Wyatt-Smith and Elizabeth Heck 


Big data and digital learning assessments... are part of the broader digital disruption 
brought about by enhanced computational and digital capacities, in terms of the 
volume, variety, and velocity of data that now circulate within countries and 
globally... Data are not inherently disruptive. It is the datafication of experience 
and the digitalization of data that are the source of disruption. 

(Wyatt-Smith et al., 2019, p. 2) 


Introduction 


This introductory chapter highlights the major issues concerning digital disruption 
in teaching and learning in schooling, and implicitly in other educational institu- 
tions. We want to do more than simply document how digital disruption is 
playing out in contemporary schooling. The chapters in the collection provide 
insightful accounts of these matters and as such contribute to contemplating 
improved social futures. Our aim in this chapter, drawing on our different 
expertise, is to open up some normative questions about how the digital might 
productively be used to construct better and more equitable schooling for all. Of 
special interest is how the digital is reshaping what it now means to be learners, 
teachers, school leaders, and policymakers, with impact on the place and the roles 
of parents, caregivers, and communities. Related are significant issues around the 
rights of the child, privacy and confidentiality, ethics, and legal issues associated 
with data use and ownership, storage, curation, and purpose. These matters also 
have high relevance to teachers, parents/carers, and communities. 

In editing this book, we conjoin our collective academic expertise in assess- 
ment and testing, education policy and governance, and the influence of the 
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digital on how learning occurs. Bringing together these domains is a potent mix: 
policy shapes the possibilities for school and teacher practices and is currently 
being heavily affected by digital disruption. Within policy frames, educational 
assessment and testing decisions always affect the work of teachers, learners, and 
school communities. This is a reality that has been intensified by digital disrup- 
tion. Adding to this are rapidly evolving digitally-based opportunities for young 
people’s learning, both in and out of school. Assessment has a particular salience 
in the work of schools as it is the main technology through which sorting and 
selection, linked to the provision of the opportunity function of modern school- 
ing, are manifested. In these ways, assessment is intensely value-laden, and to 
ensure its perceived legitimacy, it must be seen to be valid, reliable, and fair. 

Sometime ago, Bernstein (1971) wrote about the three message systems of 
schooling: curriculum, pedagogy, and evaluation (including assessment and test- 
ing). We assert that today, given the predominance of test-based accountability in 
many systems throughout the globe, the “evaluation message system” steers 
schooling systems as meta-policy. Schooling systems today have invested heavily 
in data infrastructures to manage huge volumes of data, including about school 
and individual student performance. The restructuring of schooling systems 
through new public management, and subsequently through network govern- 
ance, have enabled new non-state actors to be involved in the work of the state. 
In education, this has opened the space for edu-business, particularly EdTech 
companies, to be influential players in policy, the provision of data infrastructures, 
testing, and even curriculum (Ball, 2012). This involvement is driven by an 
overriding concern for profit, while it putatively offers the advantages of efficiency 
and dependability of testing, and a rapid return of results for use by “clients” in 
schools. These matters have been amplified in the context of the global COVID-19 
pandemic. 

It is in this context and from our cross-disciplinary approach that we seek to 
precipitate ideas and challenges, looking to raise possibilities for more productive 
and educative uses of the affordances of digital technologies and data for policy, 
teaching, assessment, and learning. We thus seek to take stock of where these 
matters are at, while acknowledging that they defy stabilization. The book addresses 
the actual and appropriate affordances of digital technology in public policy, 
including education policy, and in relation to teacher professional practices. We 
problematize: appropriate human+machine relationships in the construction and 
production of data; appropriate policymaker and teacher interpretations and uses of 
data; and appropriate design and development of data infrastructures, including data 
interoperability. These are very important considerations as the digital is restruc- 
turing contemporary schooling systems and is radically reforming teaching and 
learning. 

In what follows, we first briefly outline the broader backdrop and contexts to 
the concerns of the collection, followed by a brief account of the policy context 
that has enabled digital disruption in education, and then address the changing 
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terrain of educational assessment. We then deal with digital disruption in and 
through evaluation, followed by consideration of this disruption in relation to big 
data. The subsequent section explores digital disruption in teaching. Summaries of 
the chapters in the collection, including a suite of provocations, are then pre- 
sented prior to the summative conclusion of the chapter. The collection, in effect, 
offers the reader a contemporary pause and a moment to reflect at a critical 
juncture in human history on the future trans/formation of schooling and learn- 
ing in respect of the affordances of the digital and of data. 


Backdrop and Contexts 


We are at a critical juncture in human and societal history. Some of the tried and 
tested approaches of the past may no longer be relevant. This also appears to be 
the case in schooling. Yet, while new technologies can offer potentially new and 
very positive opportunities for education, we argue strongly that schooling and 
teacher professional practices are intrinsically relational, contextual, and cultural in 
nature and work with an ethics of care (Noddings, 2003). Further, we argue that 
this ought to remain the case in the context of digital disruption. However, a 
thorough reconceptualization of the broadest purposes of education in schools 
and the wider society, including academic and social purposes, needs to be 
addressed and adopted. In this reconceptualized frame, new technologies should 
play a vital positive role in the future of schooling. In and of themselves, how- 
ever, they are not the panacea. We are arguing here that the affordances of digital 
technologies ought to complement a new teacher professionalism. This involves 
inter alia reconceptualizing teacher evaluative expertise for discerning what data 
are useful and how they can be used to inform teaching and improve learning. 

New thinking is required about the appropriate place of the range of new 
technologies, data infrastructures, new computational capacities, and software (for 
example, big data, virtual reality, artificial intelligence, predictive analytics, 
machine learning, and individualization and personalization of learning trajec- 
tories). This is an acute need in respect of systemic policy and steering, educa- 
tional leadership, teacher professional practice, including professional judgement, 
and student learning. These matters are undoubtedly not new in deserving 
attention, although they have come to the fore in the context of the COVID-19 
pandemic. These are the core considerations of this collection, Digital disruption in 
teaching and testing. 

We note that these matters are pressing at a time of substantial social, political, 
and economic changes. They play out as growing inequality, political divides, 
economic uncertainty, mass global movements of people, greater population 
diversity, race riots, civic unrest, environmental risks and climate crises, health 
risks, rise of new nationalisms, and ethno-nationalism. These are all too familiar 
on our daily news platforms and show no signs of abating. Arguably they are 
intensifying. Perhaps paradoxically, the era is also characterized by “post-truth,” 
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“fake news,” and “anti-science” tendencies (McIntyre, 2018) in the context of a 
seemingly new positivism in data-based public policymaking (Lather, 2013). All 
of this is occurring at a time of weakening of social cohesion and significant loss 
of public confidence in civic institutions and political processes, and even of 
democracy. 

On the one hand, the new technologies could be deployed to ameliorate these 
conditions; on the other, they could exacerbate them. At the core of how these 
human futures will emerge is the key issue of appropriate human+machine 
interactions. We emphasize this point as we recognize that a technological and 
data juggernaut has been unleashed, is gaining significant power and influence, 
and is inexorably creating futures for us (rather than with us). In compiling this 
book, we adopt the position that, on a meta level, human ethics and values will 
be central to determining what sort of futures we create and the place of tech- 
nologies in that future. These matters will affect the broad societal arrangement, 
but will also play out in education policy, curriculum choices, assessment, student 
learning, and in the work of schools and teachers. 

For some time now the digital has been disrupting many aspects of contemporary 
social life. Many of these disruptive effects have been positive. Consider, for exam- 
ple, the affordances of new communication technologies (e.g., mobile phones), 
digital platforms (e.g., aspects of social media and virtual meeting spaces), and digital 
applications (generally referred to as “apps”). We also include much expanded 
computational capacity that enables the collection of unprecedented volumes of data, 
expanded analytic and predictive power, and the speed and efficiency of presenting 
new “useable” knowledge. In the early stages of digital disruption, there was a hope 
and aspiration that the new communication technologies would enable more 
democratic and inclusive participation and engagement in public and social life 
(Carpentier, 2009, 2011; Jenkins, 2006, 2009; Jenkins et al., 2013; Jenkins & Deuze, 
2008; Zuboff, 2019). There is ample evidence that digital disruption has changed 
social interactions, in some cases profoundly. Yet there are downsides to these 
developments and less evidence of the democratic benefits. 

Recent thinking has alerted us to how all of our online activity has been 
controlled by mega technology companies, and this datafication of human 
experience enables these companies to use this human data as a commodity to be 
traded and from which new profit is generated. This is what Zuboff (2019) refers 
to as “surveillance capitalism.”” We would also make the point that such sur- 
veillance techniques are utilized in market socialist countries, including the Peo- 
ple’s Republic of China. Here the Social Credit System, introduced in 2020, 
utilizes the online participation of citizens to construct individual profiles, which 
are used then by both government and business to determine what citizens will 
be able to receive, and what opportunities will be opened up or closed down for 
them. This is a manifestation of what Deleuze (1992) called “control societies,” in 
which we are always subject to evaluations. The emergence of these societies 
might also be seen as part of what the German sociologist Mau (2019) called the 
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“metric society” based on the quantification of the social. We would note that 
data from such surveillance have been “weaponized” and have become central to 
security concerns and potential risks to national governments. 

While commercialization and profit are at play in digital disruption in the 
business sector, they are also evident today in education from the earliest years, 
and are equally applicable to privatizing (outsourcing, public—private partnerships) 
and network governed schooling systems. Network governance sees non-state 
actors involved in the policy work and delivery of the state, and these non-state 
actors include national and multinational edu-businesses, individual edupreneurs, 
and philanthropic organizations (Ball & Junemann, 2012). This, in turn, involves 
the provision of costly data infrastructures, hardware and software, evidence bases, 
and systems and processes for storing and analysis of a wide range of data. To this 
we add the design of computer adaptive testing (CAT); test data analysis at 
national, sub-national, regional, and international levels; and the publication of 
reports to governments, schools, teachers, parents, and students. The private 
sector has seen in digital developments a new opportunity to remake education as 
a site for profit making. It is clear that edu-businesses are flourishing in both 
Global North and Global South contexts (Ball et al., 2017; Riep, 2014, 2019; 
Verger et al., 2016). Edu-business involvement across all stages of the policy cycle 
in education and in respect of all aspects of schooling (curriculum, pedagogy, and 
evaluation; Ball, 2012) raises matters to do with who ought to determine the 
broader purposes of schooling and the way it is delivered in societies. This 
involvement of edu-business is certainly the case in the nations of the Global 
North and is increasingly the case in the Global South as well Junemann & Ball, 
2015). 

In this book, we adopt the concept of a “socio-technical education data imagin- 
ary.” We acknowledge Jasanoff and Kim’s science and technology studies-based 
definition of “sociotechnical imaginaries”’ as referring to “collectively imagined forms 
of social life and social order” (Jasanoff & Kim, 2009, p. 120). Here, as a point of 
difference, we hyphenate “socio-technical” so as to emphasize the social construc- 
tion of new technologies, knowledge, and human involvement. At the same time, 
we acknowledge the ways in which the new technologies help constitute the new 
social and indeed spatial realities in ways that elide distance. Put simply, for us, 
hyphenating socio-technical stresses that the social and the technical are products of 
each other and that the social is constituted by human activities. Our usage then is 
also a normative one of how human/technology relationships ought to function, 
rather than be only an analytical approach; specifically, how they ought to function 
in education and, more specifically, in relation to data. 

We also focus explicitly on education and education data as they contribute to 
learning. We take such data to have primacy in reconstituting the work of edu- 
cation systems and teachers, and as having very profound effects on young peo- 
ple’s learning, their futures, and the future of social arrangements, locally, 
nationally, and globally. Here we are picking up on the datafication of schooling, 
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of teachers, and of students (see Grek et al., 2021). Our use here of “imaginary” is 
also derived from the work of Charles Taylor (2004), who argues that a social 
imaginary is a world view that underpins the understandings and actions of large 
social groups, and helps constitute the present and influences possible futures. 
This means that the social remains in a state of constant flux and that the future is 
in some ways immanent, that is, in a state of becoming. While Taylor acknowl- 
edges that social imaginaries may have links to social theory, they are different 
from social theories in their quotidian everyday character, broader adoption 
among the population, and serve to keep the social working. In effect, they have 
a normalizing function. Thus, for us, a socio-technical education data imaginary is 
about appropriate human-data relationships for shaping desirable future systems 
and practices of schooling. 

In focusing on education data, we also acknowledge there is a broader digital 
revolution going on, which some refer to as the “Fourth Industrial Revolution” 
(Pfeiffer, 2017; Schwab, 2017). This is where the new technologies are already 
having profound impacts on production, consumption, and the experiences of 
place, space, and time (Avis, 2018). In this moment in human history, we have 
the concurrent effects of a global pandemic intertwined with this Revolution in 
something of a symbiotic relationship with the two together now having an 
impact on all spheres of life. This impact is evident in education. The provoca- 
tions in this collection in Chapter 12 address the various effects of the COVID-19 
pandemic on the place of the digital in education. They collectively stress that the 
pandemic has accelerated trends already underway. 

Against this backdrop, we focus on schooling, teaching and testing, and big 
data. This provides an opening to consider what is and what ought to be con- 
tinuous and discontinuous in relation to the workings of schooling systems, the 
practices of teachers, and the role and functions of new technologies. As noted 
already, these new technologies are reconstituting schooling systems in both the 
Global North and Global South. For example, data infrastructures, as subjects of 
significant investment in many countries, created and managed by edu-business in 
many systems, are helping to constitute these systems through interoperable data 
flows (Anagnostopoulos et al., 2013; Lingard, 2019; Sellar, 2017; Sellar & Gulson, 
2019). The reach of these can be national, cross-national, and global. In respect of 
the global, they are important in the work of International Large Scale Assess- 
ments (ILSAs), such as the Organization for Economic Cooperation and Devel- 
opment’s (OECD) Program for International Student Assessment (PISA), and the 
International Association for the Evaluation of Educational Achievement’s (IEA) 
Trends in International Mathematics and Science Study (TIMSS), and the Pro- 
gram for Reading Literacy Study (PIRLS). These in turn help constitute a com- 
mensurate global space of measurement with policy effects in nations (Breakspear, 
2012). Data infrastructures are also important in enabling national testing regimes 
such as Australia’s National Assessment Program — Literacy and Numeracy 
(NAPLAN; Lingard et al., 2016; McGaw et al., 2020). 
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There is ample evidence that in most schooling systems, the appetite for more 
and more data is insatiable. Schools and systems today are awash with data, much 
of which is underutilized for policy and learning purposes. This is the playing out 
of a technical imaginary of what current and future schooling should look like 
and is one very much encouraged by large and small edu-businesses and some 
influential philanthropic organizations. There is also evidence to suggest an 
emerging cargo cult mentality among some politicians and some senior policy- 
makers that there are easy technological fixes to tame “wicked problems” in 
education (Rittel & Webber, 1973). Thus, for example, the Governor of New 
York State, Andrew Cuomo, in the context of the COVID-19 pandemic will 
work with the Bill and Melinda Gates Foundation to look at the possibility of 
schools being remade through the new technologies (see Tompkins-Stange, 2016 
on the role of philanthropy in education reform). Cuomo stated, “The old model 
of everybody goes and sits in the classroom, and the teacher is in front of that 
classroom, and teaches that class... all these buildings, all these physical class- 
rooms — why, with all the technology you have?” (Strauss, 2020, para 3). We see 
this view that technology and digital innovations will solve education problems 
and provide a utopian future for schooling as exceedingly naive. We also need to 
ask, “Who would benefit from such changes?” 

In this book, we are working with the concept of a socio-technical education 
data imaginary as enabling a different way of thinking about contemporary 
schooling and its futures, in the context of digital disruption. We reject a tech- 
nologically determinist view. Instead, we argue strongly for human leadership in 
the construction of policy, particularly in respect of the role of new technologies 
and data, and in relation to the appropriate role of data in the professional prac- 
tices of teachers and their judgements. Hotly contested here is the desired rela- 
tionship between humans+machines. For example, consider the work of 
algorithms, artificial intelligence (AI), machine learning (ML), learning analytics 
(LA), predictive analytics (PA), and virtual reality (VR). 

Algorithms are mathematical processes that help to solve problems. They are 
not a new development, though their everyday influence has grown exponen- 
tially through their centrality to computing. They permeate every aspect of 
everyday life including our use of social media, shopping, and even dating. 
Algorithms work in different ways. They remain, however, invisible to us. 
Algorithms of various kinds are necessary elements for ML and AI, but they are 
not the same thing. Both ML and AI are prevalent in discussions of big data. ML 
drives AI. Here computers bring algorithms to big data analysis and surface pat- 
terns in the data. This is a “learning process” that sets computers up for dealing 
with increasing volumes of data, and in this way ML functions as the vehicle 
driving AI. Given the current lack of big data in education, there is limited use of 
AI, but certainly future opportunities (see Gulson et al., 2018). 

Taken together, ML, AI, and algorithms have inherent potential to black box 
decision-making in a non-transparent way unprecedented in human history. 
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Black boxing, according to Rose (1999), renders “invisible and hence incon- 
testable — the complex array of judgements and decisions that go into a mea- 
surement, a scale, a number” (p. 214).? This black boxing is already well 
underway in some data uses, and potentially denies the agentic role of humans in 
decision-making in systems and schools. It seemingly offers the lure of efficiency, 
transparency, and objectivity in policy and professional decision-making, while 
neglecting and omitting human decision-making and values in the construction 
of these algorithms. Researchers such as Rose (1999) who have written about 
black boxing in relation to governance by numbers have been referring to the 
technical work that goes into the construction of statistical numbers, literally state 
numbers and the like. We would suggest that the black boxing associated with 
algorithms hides in even deeper ways the multiple processes they utilize so as to 
provide insights from large data sets and those that are longitudinal and linked 
with other data sets. Those who have written about the black box phenomenon 
in assessment have focused on assessment for learning, which could be considered 
as part of the secret garden of teachers’ work (Black & Wiliam, 1998). Some 
assessment theorists have called for the lifting of the lid on the black box of tea- 
cher judgement in assessment practices to give greater transparency to this ele- 
ment of teacher expertise (Wyatt-Smith & Adie, 2019). In this chapter, we 
advocate for the important place of human judgement and expertise in relation to 
the construction and production of data and the related use of algorithms. 

Within this frame, the affordances of the new technologies need to be 
acknowledged, along with the place of fit-for-purpose data in good curriculum 
design, rich pedagogies, and assessment practices. However, there are significant 
issues in how we move from the present to a future that will be even more 
technologically mediated. Among these issues, perhaps the most pressing, is what 
should be held onto from the past and present, and what should be made entirely 
new for the future. Currently, humankind does not have a clear line of sight to 
the future. Writing about the effects of the end of the Cold War and related 
intellectual challenges to Enlightenment values, Laidi (1998) argued that “the 
need to project ourselves into the future” has never been greater, but “we have 
never been so poorly armed on the conceptual front to conceive this future” (p. 
1). Thirty years after the end of the Cold War and in the context of digital dis- 
ruption and the negative contemporary political realities alluded to earlier, this 
observation appears even more applicable now than when Laidi made his obser- 
vation more than 20 years ago. 

Yet we acknowledge that technologies and data will play an even greater role 
in how we live, work, and what counts as meaningful civic contributions in the 
context of digital governance. In that context, we would still argue that teaching, 
like medicine, is an inherently relational practice, and this ought to continue, 
whatever the technological future holds. At issue, therefore, are the knowledge, 
skills, expertise, and dispositions that will be necessary for a productive future 
teaching profession. 
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Policy and Assessment Frames 


Education Policy 


The digital disruption in education that is the focus of this collection has been 
aided and abetted over the last three decades by new practices of statecraft and a 
restructured (post)bureaucratic state. Following the end of the Cold War and the 
rapid globalization of the economy, the bureaucratic state in most nations was 
restructured to accompany the neoliberal framing of the global economy. State 
restructuring and neoliberalism were not the same thing; they had different 
rationales, but the new state structure sat comfortably with the neoliberal frame of 
the global economy, which resulted from nations breaking down tariff barriers 
and encouraging global free trade. The restructured state, first under New Public 
Management and subsequently through network governance, occurred against 
the back drop of the neoliberal prioritizing of the market over the state, indivi- 
dual responsibility over state welfare, and a seeming view that the private sector 
was more efficient and effective at achieving its policy goals, than was the 
bureaucratic state, which was hierarchical in character and framed by technical- 
rationalist rule following. The classical Weberian bureaucratic state structure was 
deemed to be too sclerotic for the fast character of global capitalism and economy 
and also in respect of “fast policymaking” (Peck & Theodore, 2015). This wit- 
nessed the use by the restructured state of public-private partnerships as a way of 
delivering state services and the opening up of other opportunities for the invol- 
vement of edu-businesses and profit making in education (Ball, 2007). 

New Public Management (NPM) sought to thin out the bureaucracy and steer 
through outcome measures rather than through input measures. Here we can see 
the move from the Keynesian policy frame of extensive state spending and 
steering to a more frugal state and thinned-out bureaucracy, emphasizing out- 
comes through the creation of performance indicators and the like. It is in that 
context that the digital disruption and enhanced computational capacities enabled 
a new governing by numbers, with data becoming more important in state policy 
steering and in top-down modes of accountability. This saw the bureaucratic 
center setting broad goals, steering at a distance, with the frontline, street-level 
deliverers of these policies held to account through outcome measures. 

We need to acknowledge that numbers as statistics, literally state numbers, have 
long been central to the functioning of modern bureaucracy and democracy 
(Desrosieres, 1998). What is different now is the enhanced computational capa- 
cities and the greater ability and desire to “datafy” and quantify the social (Mau, 
2019; Zuboff, 2019). Large amounts of state data required data infrastructures, 
which were often provided by the private sector, as were the standards according 
to which they operated (Sellar, 2017). Indeed, the restructured state opened up 
many opportunities for the private sector to work with and for the state for 
profit-making in relation to data work, which became more important. The place 
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of data in the governance of education has moved beyond the initial creation of 
“infrastructures of accountability” (Anagnostopoulos et al., 2013) created to manage 
test-data towards “digital education governance” (Williamson, 2017, p. 94) with 
data playing an enhanced role in the functioning of school systems. 

This state restructuring affected education systems, as did neoliberal ideology. 
There was an attempt to create quasi-markets in schooling. These markets were 
instantiated through the implementation of top-down, test-based forms of account- 
ability, with comparative test scores putatively enabling parental choice of schooling. 
In many nations, census national testing was introduced and functioned as a com- 
plement to international large-scale assessments (ILSAs). Numbers and data became 
ever more important in the work of systems and schooling. Comparison of perfor- 
mance was now central to governance; here Mau (2019) has observed that “data 
create a comparative panopticon with diverse visual axes of numeric comparison” 
(p. 159). Regarding this mode of accountability, Ranson (2003) argued that this 
mode, with schools (and systems) being held to account for a range of test scores and 
performance measures, was actually becoming the system. Related, Ozga (2009) has 
argued persuasively that schooling is now governed through data. 

NPM subsequently transitioned to network governance (Ball & Junemann, 
2012). With network governance, private sector actors (edu-businesses and 
EdTech companies, also philanthropists) entered the state, its structures and 
practices, and had influence across all aspects of the policy cycle in education. 
These included agenda setting, through policy production, to policy imple- 
mentation and evaluation, with potential impact on curriculum, pedagogy, 
assessment, and reporting (Ball, 2012). State restructuring also saw one-line bud- 
gets devolved to schools with school autonomy enabling school-level purchase of 
education apps and other digital technologies from the private sector. The recent 
rise of nationalist backlashes against globalization have begun to have an impact in 
politics and policies, but the state restructurings in education outlined here have 
remained in place (Lingard, 2021). 


Educational assessment 


In this section, we consider debates in educational assessment and the opportu- 
nities the digital opens in relation to these debates. The debates are far from new 
but take on new salience in the current digital era, noting Delandshere’s (2002) 
point that there the field lacks a general theoretical position that connects assess- 
ment to “meaning making.” Over the 20th century and to the present, assess- 
ment writers have continued to concentrate on issues of validity*, reliability”, 
standards, and accountability, while human judgement, decision-making, and 
evaluative expertise have received less sustained attention (Cooksey et al., 2007). 
Limited attention has been paid to the affordances of digital technologies beyond 
the focus on data infrastructures, standardized testing, scoring, and reporting, lar- 
gely driven by systems and private testing companies. 
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Historically, assessment has served two main purposes with distinct goals; first, 
summative assessment serving the goals of measurement and reporting, and 
second, formative assessment serving the goals of learning improvement (Harlen, 
2005a; Sadler, 1989; Wiliam & Black, 1996). Summative assessment typically 
occurs at the end of a designated phase of study, say at the end of a term or 
semester, or at the exit from a course of study. It is primarily concerned with 
measuring individual achievement or competence and securing high reliability. 
Examinations have been the primary mode for summative assessment. The tasks 
included in examinations are necessarily limited, being subject to the conditions 
of restricted time for student completion. Examinations also prioritize rater- and 
inter-rater reliability with less emphasis on claims to validity and authenticity of 
assessments. 

Formative assessment and the evidence of learning that it produces are most 
commonly collected by the classroom teacher (Black & Wiliam, 1998). Formative 
assessment and feedback are intended to occur as part of teaching and to serve the 
goals of improving learning (Black & Wiliam, 1998; Shepherd, 2000). Formative 
assessment is concerned with evidence for informing decisions about the student’s 
current level of performance; the desired and realistically attainable next level of 
performance; and the strategies necessary to close the gap between the two 
(Ramaprasad, 1983; Sadler, 1989, 1998). In short, it serves to inform next steps 
teaching and learning, whereas summative assessment is not intended, generally 
speaking, to have a future application. 

In formative assessment, validity and authenticity have been priorities. The 
alignment of curriculum, pedagogy, and assessment is also a feature along with 
clarity about expectations of quality. However, one issue for enhancing the status 
of formative assessment has been the reported problems of the dependability of 
teacher judgement. Human judgement in assessment practices has continued to 
be construed as largely subject to the influence of random error, bias, and even 
whimsy. Interestingly, teacher judgement has not been widely trusted by systems, 
though research shows that when calibrated, teacher judgement can be rendered 
reliable (Harlen, 2005b). Research also shows that dependability of teacher jud- 
gement can be achieved through explicitly stated criteria and standards, and 
moderation practices where teachers come together to share samples and apply 
standards supported by “cognitive commentaries” (Wyatt-Smith et al., 2010) in 
which the deep structures of judgement are revealed. The commentaries serve to 
reveal the appraisal processes used by teachers to arrive at an overall grade, 
including complexities of trade-offs that they considered in combining both the 
stronger and weaker aspects of performance. These research insights into judge- 
ment have informed initiatives in feedback, including voice recorded feedback on 
the application of criteria in the course of scoring and commentary to inform 
future learning. However, despite these developments, examinations have 
retained the highest status as producing trusted and fair measures typically regar- 
ded as objective assessment. 
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Historical boundaries between formative and summative assessment are well 
established. The teacher has been the primary agent in the former, and in the 
latter, the state and education jurisdictions have prescribed requirements for 
examinations, scoring, and reporting. The boundary has served to ensure that 
examinations for summative purposes have been the province of examination 
boards and testing companies at system level, committed to demonstrating 
system and sector accountability. The formative-summative boundary has held 
in most countries. This became crystal clear in the turbulence caused by the 
pandemic when terminal examinations could not be scheduled; systems 
needed to maintain a pipeline of students through to graduation from 
schooling and to develop radically different means for assessing and reporting 
student achievement. 

In recent times, there has been a laser-like focus on improved learning with 
schools coming under the concerted gaze (panopticon) of the state and testing 
companies. Formative assessment has now moved to be the new domain of soft- 
ware developers and testing companies. Discourse has shifted to assessments 
online, at the point of need, serving to guide teacher decision-making, and cus- 
tomizing monitoring of learning and learner growth. This has been fueled in part 
by the appeal of digital technologies to enable easy delivery of assessments on a 
continuing basis, fulfilling the purposes of formative assessment as mentioned 
earlier. There is already the imperative in some countries to collect such mon- 
itoring data online and in this way, extend the gaze directly into schools and 
classrooms. In the digital disruption era, the boundary between formative and 
summative has blurred and accountability is being reconceived as extending to 
collecting and storing online point-in-time data on student learning, teaching 
quality, and school and system performance. Added to this is the formative use of 
national and international testing data, even though by design, the tests are not 
constructed for these purposes. 

This raises the key question of where change in education in general and 
assessment specifically comes from. The answer seems to be clear: change is 
coming to education from agents with expertise outside schooling. These include 
edu-businesses, philanthropic organizations, private entrepreneurs with capital 
investment in schooling (e.g., charter schools), testing companies, and more 
recently, EdTech companies, systems designers, and data analysts. Their collective 
incursions reflect that education is now a prized marketplace and how assessment 
for both formative and summative purposes has become big business. Somewhat 
ironically, these moves occurred at a point when education research confirmed 
the value of formative assessment in teachers’ hands (Black & Wiliam, 1998; 
Earle, 2003), but at the same time, revealed that formative assessment had not 
worked in schools (Baird et al., 2017; Booth, 2017). This possibly reflects the 
historical dominance of external examinations and the failure to develop evalua- 
tive expertise among teachers, especially regarding teacher judgement and the use 
of standards (Wyatt-Smith & Adie, 2019). 
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Working Together — Policy and Assessment Frames in the Digital 


To circle back, the new practices of statecraft, along with exponentially evolving 
dependence on numbers and data, in education policy, and in the work of sys- 
tems and schools, have opened the space for for-profit edu-businesses and 
EdTech companies to take on ever increasing roles in them. What we have seen 
are moves towards an emergent mode of “digital education governance,” a con- 
cept that “registers the displacement of educational governance to new digitized 
sites of expertise in data collection and analysis” (Williamson, 2017, p. 94). This 
form of governance “also acknowledges the role of digital software, code, algo- 
rithms in governing and guiding the conduct of diverse educational actors and 
institutions” (p. 94). Digital governance has serious implications for education 
budgets, the economics of education, and democracy more generally. 

We assert that much of the digital development in the fields of assessment and 
testing, driven by testing companies, often in partnership with governments, has 
prioritized the psychometric properties and efficiencies of testing and scoring over 
a fulsome engagement with the scholarly corpus of literatures on assessment and 
evaluation, which we have succinctly traversed in the subsection above. Over- 
coming this divide would demand collaborations and a sharing of expertise across 
numerous fields, including digital architects; learning engineers; systems designers; 
psychometricians; education data scientists; and curriculum, pedagogy, and 
assessment experts including teacher practitioners. We next consider the impact of 
the digital on assessment and testing. 


Digital Disruption in and through Assessment and Testing 


Globally, in relation to digital disruption, most current debates in education are 
concerned with the role of tests and test data, and how this will assist teachers and 
policymakers. Put simply, digital learning assessments involve the design of standar- 
dized tests in an online format. For this to occur, optimally, they are “born digital” 
(McGaw et al., 2020, p. 126). Additionally, they are structured in a way to enable 
rapid computer scoring, a quick turnaround of results and feedback, and subsequent 
analysis at system and local levels. Here we make a distinction between digitizing tests 
and digitalizing them, with the former simply the reproduction of the paper and 
pencil test online. In stark contrast, with the digitalization of standardized tests, the 
online format recontextualizes and rearticulates the test. This means the test appears 
differently, students engage with the test differently, and the opportunities are there 
for multimodal design of items. Moving to digitalized assessments requires much 
more than simply reproducing a print text in an online environment. Such learning 
assessments are thus freed from the constraints of paper and pencil tests and framed by 
the affordances of the technology. 

Computer Adaptive Testing (CAT) is the contemporary manifestation of 
assessments born digital. To date, CATs have been developed utilizing a detailed 
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item bank. CATs are designed so as to adapt the test to the abilities of those 
taking the test, so that not all test takers answer the same sets of items. This is 
called “branching”: test takers’ engagement with the test is tailored to their per- 
formance on the items at predetermined junctures in the test design. This is 
where algorithms are important; it is an algorithm that selects the items for test 
takers in response to their success or not in answering initial or subsequent ques- 
tions. Figure 1.1 presents an example of how branching works in computer 
adaptive testing, and Figure 1.2 represents how branching manifests in the 
national testing of literacy and numeracy in Australia for those students who take 
the test online. 

There is no doubt that the development of digitalized assessments brings new 
challenges for test design, analytic capacity, scoring, and reporting mechanisms for 
systems, schools, and teachers. We note that a move from paper and pencil to 
online testing requires a new time series and the selection of new anchor items. 
Anchor items are important for tracking performance over time. The fiscal capa- 
city, expertise, and professional competence to meet these challenges vary widely, 
even within a country. In a context in which governments are both demanding 
and dependent on a supply of education data for accountability purposes, these 
capacity shortfalls have opened the space for the outsourcing to for-profit edu- 
businesses to provide their costly services. These costs can include those associated 
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FIGURE 1.1 Test tailored design with branching 
Source: Australian Curriculum, Assessment, and Reporting Authority (ACARA; 2014). 
Reproduced with permission.° 
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FIGURE 1.2 Branching structure of NAPLAN online literacy and numeracy tests 
Source: Australian Curriculum, Assessment, and Reporting Authority (ACARA; 2020). 
Reproduced with permission.” 


with item development for online delivery, administration of the test, secure test 
data storage, security of test items for use in subsequent tests to potentially enable 
longitudinal analysis, and the application of predictive analytics. 

Here we recognize that this set of issues has serious implications for tests and 
test items, and for the provision of feedback and reporting on the performance of 
cohorts, special at-risk groups, and individuals. We also note that there is very 
little published cost-benefit analysis undertaken by governments to examine the 
costs and benefits of census and sample standardized testing. Cost-benefit analysis 
is common practice in other domains of government activities. 

Within this set of complex issues, writing assessment is well recognized to be 
perhaps the most difficult domain in designing standardized assessments, with 
these issues intensified in attempts to digitalize such assessments, including the 
scoring of them. To illustrate, in their current forms, PISA, PIRLS, and TIMMS 
are among the high-status international large-scale testing programs that do not 
include tests of writing. Assessing writing in domain areas lies generally beyond 
large-scale learning assessments, and inter-country assessment of writing is recog- 
nized to be in its infancy (McGaw et al., 2020; United Nations Educational Sci- 
entific and Cultural Organization (UNESCO, 2019). At the point of moving 
writing to online testing, it is interesting to note how few countries have installed 
this to date (see McGaw et al., 2020, p. 163). 

There are numerous issues with the testing of writing, whether it is completed 
online or with traditional technologies. This is the case for a range of reasons. 
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There are, for instance, decisions to be taken about the forms/genres of writing 
to be assessed, the related curriculum areas in which the writing is situated, the 
conditions under which the students undertake the test, and provision of time for 
planning, drafting, and revising. Added to this is how to develop digital stimuli to 
engage young people in an authentic writing test. There are also complex questions 
regarding assessment criteria for scoring, the potential influence of co-dependencies 
among criteria and how these can affect scoring, and the means by which reliability 
can be demonstrated. Additionally, there are questions to do with assessment stan- 
dards and moderation across participating jurisdictions and importantly, considera- 
tions of the role of the profession in test design, implementation, including scoring 
and moderation, and reporting. 

Among these is perhaps the biggest question of all: “Is machine scoring 
equivalent to human scoring of writing assessment?” At its core, this question calls 
for careful research into the validity of automated scoring systems related to the 
validity of test content and response processes (Bridgeman & Ramineni, 2017; 
Shermis, 2014). Moves to machine scoring of writing have triggered deep skep- 
ticism among citizens and the teaching profession regarding the prospects for fair 
and informed assessment of writing by machines. The press for online testing of 
writing is nevertheless evident in at least some contexts. The Australian Govern- 
ment is one example, calling for all states to move to national numeracy and lit- 
eracy testing online, including writing, as soon as possible. This brings to the fore 
priorities concerning moving tests online or improving the quality of the tests. 

Going beyond writing, the scope of testing is a further important consideration 
in the management and delivery of computer adaptive testing. In census testing, 
whole cohorts participate, as distinct from sample testing, where a purposive 
cohort is drawn. We note that these two types of testing serve different purposes, 
and in census testing in particular, resource implications are significant. Such 
testing is dependent on an important set of conditions for effective implementa- 
tion. These include: the necessary digital infrastructure at school and site levels, 
necessary bandwidth and stability of online delivery, essential provision of hard- 
ware and software, and student keyboarding fluency and typing competence. The 
last of these cannot be under-estimated; such competence reduces the cognitive 
demands of responding in online test environments. Additionally, we note the 
challenges of ensuring sustained Internet connections and secure supply of power 
in some nations, particularly in the Global South. 

The next generation of digitalized assessment testing has been characterized as 
“immersive assessments” (Wyatt-Smith et al., 2019). Figure 1.3 shows that these 
are an evolution from Computer-based Testing (CBT), through Computer 
Adaptive Testing (CAT), to Immersive Assessments (IA). The features of these 
digitally mediated assessments are identified in Figure 1.3. Immersive assessments 
rely on augmented reality (AR), virtual reality (VR), mixed reality (MR), and 
extended reality (XR). Immersive assessments also rely on artificial intelligence 
and 3D developments that focus sharply on the cognitive, experiential, and 
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FIGURE 1.3 The evolution of digitally-mediated ways to assess student progress 
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Source: Wyatt-Smith, C., Lingard, B., & Heck, E. (2019). Digital learning assessments 
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sensory potential of digital learning assessments that include psycho-informatics. 
Williamson (2019) suggests that, “Psycho-informatics is based on the application 
of computer science techniques to psychological tracking, measurement and 
analysis of behaviors, emotions, personality traits, attitudes, cognition and abil- 
ities” (p. 66). He goes on to suggest that psycho-informatics “makes use of 
behavioral data sources and analytical platforms employing techniques from data 
mining and machine learning to detect, characterize and classify behavioral pat- 
terns and trends” (p. 66). 

To succinctly define the constituent parts of immersive assessments, augmented 
reality (AR) uses the real world environment augmented by computer-generated 
input (e.g., sounds, global positioning systems (GPS), touch); virtual reality (VR) 
immerses participants in completely virtual environments (for VR in curriculum 
and pedagogy, see Southgate, 2020); and mixed reality (MR), sometimes called 
hybrid reality, merges real and virtual worlds where the physical and the digital 
co-exist and interact in real time. Extended reality (XR) is where real and virtual 
environments are combined, and human+machine interactions are generated by 
computer technology and wearables. XR is an umbrella concept that brings AR, 
VR, and MR together. 

Immersive assessments have the benefits of computer-based assessment of 
scoring efficiency and prompt feedback. However, they have potential additional 
benefits of creating virtual realities that enable an authenticity in assessment in 
ways not previously possible. We are not suggesting, though, that all knowledge 
forms lend themselves equally well to these immersive assessments. Elsewhere we 
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have argued that the evolution, represented in Figure 1.3, is not intended to 
suggest a linear progression, and is also not intended to imply that all schooling 
systems around the globe should be moving to immersive assessments at this time 
(Wyatt-Smith et al., 2019). Readers will be well aware, however, that simulated 
reality is central to computer gaming with which many young people are familiar. 
However, while simulations have also been utilized in some instructional 
domains, such as the training of pilots, military personnel, dentists, surgeons, and 
other health care professionals, simulated reality has had limited systematic use in 
schooling to date. We note that there are multiple potential positive uses of 
simulated reality in classrooms across multiple subject areas, stretching from Sci- 
ence through the Arts and Physical Education. There is also work underway 
developing virtual reality classrooms in which students are immersed in the virtual 
realities of what they are studying (Southgate, 2020). The lack of uptake of 
simulated reality is even more the case in respect of assessment. Rich immersive 
assessments are costly to design and costly to implement. There is thus the 
inevitable situation of the haves and have nots in relation to these developments: 
those who have opportunities to access such advanced assessment practices, and 
the knowledge and expertise needed to complete them, and those vulnerable 
populations with no such access, resulting in a widening of the gap in opportu- 
nities and learning. This discussion suggests that digitalized assessments have the 
potential to exacerbate inequalities in the experience of schooling across different 
populations and across nations. 

For most readers, the experience of assessment has involved undertaking a 
prescribed, non-negotiable task, alone and unaided, under time restricted non- 
negotiable conditions, with limited or no access to human or material resources. 
What we have been considering here in relation to the evolution and disruption 
of assessment and learning are challenges to the traditional, taken for granted 
assumptions about assessment. Moreover, this exogenous pressure to change due 
to digital disruption demands new ways of assessing and new ways of demon- 
strating accountability. 

The COVID-19 phenomenon has taught us that the tried and trusted ways of 
bringing young people together to sit terminal examinations in large halls, with 
invigilators patrolling the desks, are not the only ways. In this context, systems 
and leaders have been forced to find new ways to assess and rank students. There 
is a deep irony here in the context of COVID-19 and schooling from home; at 
least in some countries, there has emerged a new respect for teachers and their 
professional judgements and also the recognition of the complexities of teachers’ 
work. Yet we note the fiasco in 2020 in England in relation to results generated 
by algorithms for GCSE and A-level students who were not able to sit their 
examinations due to the pandemic. The fiasco was caused by deep concern 
reported by parents, students, and the broader community about the veracity and 
fairness of the algorithmically derived results based on historical data. The Prime 
Minister blamed the fiasco on a “mutant algorithm” (Poole, 2020, para. 1). Of 
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interest in this book is how this example illustrates the way in which past history 
of performance became hard wired into the algorithm, which had the effect of 
reproducing structural inequalities, thereby disadvantaging high performing stu- 
dents in schools that had historically performed poorly. This brought into clear 
sight the ways in which schooling systems, especially through assessment, have 
tended to reproduce, rather than ameliorate, structural inequalities. Interestingly, 
some activists have argued that algorithms used for public policy purposes should 
be subject to extant anti-discrimination policy. 


Digital Disruption in and through Big Data 


We begin this section by offering a working definition of big data and then 
unpack it. Big data is one of those concepts with “terminological simplicity that 
marks highly contested territory” with “data” recognized as “the most elusive 
term of all” (Borgman, 2019, p. 1). While acknowledging the elusive nature of 
the concept of data, we are taking it here to refer to “numerical information in 
digital formats” (Sellar, 2017, p. 341). Attempts have been made to define the 
combination of data + big, including by Williamson (2017), who formulated the 
following technical definition of big data drawing on the cognate literature: 


In technical terms, big data refers to data sets that are huge in volume (at the 
scale of petabytes, exabytes and zettabytes); highly diverse in type and nature; 
generated continuously at great velocity in or near real-time; exhaustive in 
scope (enabling the capture of entire populations — or “n = all” — rather than 
sampled selections); fine-grained in resolution at the level of indexing individual 
units; combinable with other networks of datasets; and flexible and scalable 
enough for new fields to be added and to expand in size rapidly. (p. 32) 


This extract captures the commonly referred to characteristics of big data as 
volume (necessitating cloud storage), variety, and velocity. Drawing on the lit- 
erature, De Mauro et al. (2016) also defined big data in a way that highlights that 
it is an asset. In so doing, they suggest its potential for transformation and in turn, 
value-added status and monetization. These authors wrote that “big data is the 
information asset characterised by such High Volume, Velocity and Variety to 
require specific Technology and Analytical Methods for its transformation into 
Value” (p. 131). This later point relates to the usage of big data analytics “for 
intelligent decision-making” (Hilbert, 2016, p. 139), involving predictive analy- 
tics. There are interesting considerations here in respect of how these matters 
construct the temporal, particularly relationships between the past, present, and 
predicted futures and on what basis (see Webb et al., 2019). 

Such definitions provide an opening to consider various matters relating to data 
that are distinguishable in nature and kind from data themselves. These include, 
for example, the necessary computational and infrastructural capacity to collect 
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and store large volumes of data. These preconditions for working with data rou- 
tinely involve substantial costs. Dependent on the scale and nature of the data, 
these preconditions open up issues of network governance and related opportu- 
nities for the private sector to bring profit-making motives to how the data are 
processed and even on-selling opportunities. 

According to Sellar (2017), a data infrastructure “can be understood as an 
active and changing platform for storing, sharing and consuming data across net- 
worked technologies” (p. 345). Such infrastructures rely on technical, software, 
and hardware capacities to facilitate system and data interoperability. Sellar also 
identified that a number of interoperability standards have been developed, an 
example being Microsoft’s Schools Interoperability Framework (SIF), which was 
developed in conjunction with 18 other software companies and the US Software 
and Information Industry Association (Sellar, 2017). From 2003, the US Federal 
Department of Education became involved in these developments. In Australia, 
governments, in conjunction with the EdTech industry in 2009, moved to 
develop the National Schools Interoperability Program (NSIP) to ensure the 
interoperability of all data from all systems and all schools across Australia. 

Analytical capacity to design algorithms, applied to multiple data sets, is yet 
another issue, together with the ways in which the algorithms are used in pol- 
icymaking. For example, the continuous collection of data and how big data are 
constituted over time also raise questions of privacy, confidentiality, and rights of 
students, teachers, and carers. As boyd and Crawford (2012) have argued, while 
big data can provide solutions to problems, they pose issues about privacy. These 


” 


issues have been referred to as “dataveillance,” a concept taken to denote the 
collection and monitoring of personal data (i.e., data of a range of types relating 
to individual identity and group membership, and even health records). These 
privacy issues become even more significant when EdTech companies are 
involved in the collection, storage, and analysis of data, with the potential to on- 
sell to third parties. Usually, such uses occur without the consent of those whose 
data have been collected. This is an example of network governance at work 
with private, non-state actors involved with the state in developing policy. 
There is growing concern, even cynicism, about government and system use 
of data. Lingard et al. (2017) identified that Australian teachers were deeply 
concerned about student data being in private hands. Further, in the USA, there 
was the collapse of the InBloom initiative in a group of Eastern states, which had 
been supported by philanthropic money. In response to then President Obama’s 
Race to the Top Initiative (RTTT), the aim was to create data infrastructures 
and interoperability between multiple data sets, across government portfolios 
and across states. The collapse reportedly occurred as a result of organized par- 
ental opposition to the involvement of private companies, data privacy issues, 
and the potential for student data to be on-sold to third parties for profit 
(Bulger et al., 2017; Lingard, 2019). In Japan, data privacy has also been a 
concern to the point where legislation ensures data collected by governments 
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can only be used for the explicitly defined purposes for which they were col- 
lected (Takayama & Lingard, 2019). 

With big data, using data linkage, various extant data sets can be integrated, 
and algorithms used to produce “patterns” and other insights can make visible 
solutions to seemingly intractable problems. Drawing on Mayer-Schonberger and 
Cukier (2013, 2014), this means reducing emphasis on causality and a turn to 
correlations. This focus on correlations raises the prospect that we will know what 
has occurred, but not why. This has implications for subsequent stages of decision- 
making. We stress yet again that human decision-making is the necessary element 
to interpret such meaning from data and what it may mean for future actions. 
This is certainly the case in relation to education and student learning. 

So, how do data and specifically big data apply and work in education? Cope 
and Kalantzis (2016) define these concepts in the following ways: 


(a) The purposeful or incidental recording of interactions in digitally-medi- 
ated, cloud-interconnected learning environments; (b) the large, varied, 
immediately available and persistent datasets generated; (c) the analysis and 
presentation of data generated for the purposes of learner and teacher feed- 
back, institutional accountability, educational software design, learning 
resource development, and educational research. (p. 197) 


The above definition opens wide the vista of students individually and in small 
groups working continuously on computer-based curricula, pedagogy, and assess- 
ment. This is the individualization, personalization, and customization aspect of big 
data as it comes into, and attempts to service the classroom, be it in schools or uni- 
versities. There is the real prospect of schooling to be recast to prioritize what can be 
produced and presented online. Here we acknowledge that such data have the 
potential to “recreate the world,” rather than simply represent it (Espeland & Sauder, 
2007). Rose (1999) suggests that while numbers and data represent reality, they are 
also simultaneously “inscription devices” that help constitute that reality. 

When this occurs, the broader social and citizenship goals of schooling could 
be considerably downplayed and recast. The risk is that the message systems 
(curriculum, pedagogy, and evaluation; Bernstein, 1971) in online format become 
standardized and reductive and come to redefine schooling. In turn, we see here 
the possibility of the message systems being constituted and controlled by edu- 
businesses and for profit. Sellar and Hogan (2019) point to the possibility that 
personalized online schooling might elide considerations of the social purposes of 
schooling, including for example, learning to collaborate and work in teams. We 
note that there has been parental opposition to these developments as evidenced 
in the opt-out movement in the USA, which, while mainly focused on high- 
stakes testing, has also been concerned over edu-business control of the message 
systems and of schooling reduced to students working all day on computers 
(Lingard & Hursh, 2019). 
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All this points to pressing new questions for our times: Who ought to develop 
curricula in schools in democratic societies? What should be the role of the tea- 
cher (or knowledgeable other) in relation to computer-based scripted curriculum, 
pedagogy, and evaluation? Do the answers change in low-income countries, 
when it may not be possible to produce professionally educated teachers in the 
necessary numbers? 

There are very few national and cross-national large-scale learning assessments 
that might be defined as big data. The case of Australia’s National Assessment 
Program Literacy and Numeracy (NAPLAN) illustrates how little big data there 
might actually be in extant school assessments. This is despite the fact that 
NAPLAN is a census test involving full cohorts in Years 3, 5, 7, and 9 at the time 
of writing. Similarly, international large-scale assessments including PISA, 
TIMMS, and PIRLS are sample tests and do not currently generate big data. 

Edu-businesses have been among the main advocates of a big data approach to 
reforming schooling systems and schools (Hill & Barber, 2014; Hursh, 2017; 
Sellar & Hogan, 2019; Williamson, 2017). Perhaps it is therefore not surprising to 
see the upswing of engagement by edu-businesses in what has previously been 
the domain of the classroom teacher, namely learning assessments (sometimes 
called assessment for enabling learning or formative assessment). The primary 
purposes of such formative assessments have been to assist with teachers’ instruc- 
tional decisions. Has the child learned the concept? Is the child ready to proceed 
to next-step learning and teaching? What is the child’s current level of perfor- 
mance; what is the desired level, and importantly, what are the necessary actions 
to close the gap across the zone of proximal development (between the current 
and desired levels) (Ramaprasad, 1983; Sadler, 1989; Vygotsky, 1997). 

To date, learning assessments have been the domain where the teacher’s cur- 
riculum, pedagogical and evaluative expertise have been the source of their pro- 
fessional decision-making and judgement. In contemporary times, however, in 
high-income countries, this is a relatively new space for schools and parents/carers 
to purchase formative assessment apps and intelligent tutoring systems for online 
learning (e.g., Reading Eggs, Mathseeds, Mathletics, and Quizlet), as well as apps 
for behavior management (e.g., Class Dojo®). There are also some of these in 
middle-income countries such as India (e.g., BYJU’s — The Learning App’). 
These formative assessment apps are effectively creating new markets and, in turn, 
catalyze a big data disposition: insights from built-in analytics can be readily 
deployed by private companies to improve the apps to increase their effectiveness 
and attractiveness for purchase. Far less overt are questions of ownership of data 
collected about teachers and students, and further, how these relate to bigger 
questions about data ethics, matters of consent, and issues of data privacy. 

Whether and how consent is given to the collection of such data and who 
owns it are significant and complex issues. We argue that this is particularly the 
case when the mode of assessment is conducted by private actors, and done 
within, and sometimes contracted by governments, schooling systems or schools, 
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and other administrative agencies relating to the teaching workforce. The impli- 
cations of this commercial arrangement, supported by hardware and software, can 
be significant when they manifest as achievement targets and track performances 
of students in ways that are not formally authorized by the students and in some 
cases, even school leadership: surveillance schooling has potentially arrived. 

A report prepared for the European Commission explored issues to do with 
data privacy in relation to big data (Berendt et al., 2017). The report recom- 
mended that data protection and data privacy both be explicitly incorporated into 
the design of big data systems. Agreement on such measures, however, especially 
as they relate to ethics and integrity protocols, remains a work in progress. Zuboff 
(2019) has made a similar point about the potential taming of big data through 
legislative and legal frameworks in relation to privacy. 

The research on schooling systems and leadership suggests that schools in several 
countries are awash with data (Landri, 2018; Williamson, 2017). The problem 
therefore is not that schools, teachers, and school leaders do not have sufficient data 
to inform their practices and decision-making. On the contrary, they have such a 
constant deluge of data that the onerous task is to discern fitness-for-purpose of the 
data they have. Adding to these demands are what Polonetsky and Jerome (2014) 
have identified in their US report as ethical and privacy issues associated with the 
rapidly expanding data in US schools. They explored legislative frameworks and 
suggested that these appear inadequate to the task at this point, especially with the 
enhanced involvement of EdTech companies and other edu-businesses in respect of 
data work. Undoubtedly, there are complex legal and ethical issues here concerning 
data ownership. The US report exposed the intricacies of consent by students and 
parents in respect of various data sets. These matters remain in their infancy in rela- 
tion to legislation in education and the rights of the child. These issues are 
undoubtedly becoming more significant to the public and legislators, indicated by 
privacy and usage concerns about the data that citizens readily surrender to social 
media (e.g., Facebook, Twitter, Instagram; Zuboff, 2019). According to Williamson 
(2017), “considerable unresolved concerns remain about the adequacy of con- 
temporary student privacy and data protection policies and frameworks in relation 
to the rise of educational data science practices of big data analytics and data mining” 
(p. 121), including the local use of data dashboards. The era of data walls (as material 
artifacts) has been replaced by digital displays, including on the big screens in schools, 
with little systematic research on how they are being used, and with what educative 
effects (Adie et al., 2020; Harris et al., 2020). We suggest exactly the same about 
issues of data ownership and the rights of the child/student/parents in the current 
moment of digital disruption to schooling. 


Digital Disruption in Teaching 


Elsewhere we have identified that digital disruption may precipitate a rapid 
deprofessionalization of teachers in low-, middle-, and high-income countries 
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(Wyatt-Smith et al., 2019). In some low-income countries, schooling is provided 
by unqualified teachers and machines, using scripted curriculum, pedagogy, and 
assessment provided by for-profit edu-businesses (Junemann & Ball, 2015; Riep, 
2015, 2017). In some middle- to high-income countries, teacher e-readers can be 
readily connected to smartphones with resultant data moving from teacher com- 
puters to storage in the cloud to frame next step teaching. At the very core of 
efforts to maximize educative potentials of digital technologies is teacher exper- 
tise. In addition to the necessary teacher expertise, policy also needs to be cog- 
nizant of digital divides within and across nations, so as to avoid further 
entrenching inequalities in and through schooling, which have been growing 
across the past two decades (Chmielewski, 2019). 


Digital Literacy and Data Literacy for Teachers 


Teacher professionalism encompasses various professional knowledges and practices 
(e.g., in relation to curriculum, pedagogy, and evaluation) that are paramount to the 
development of learning for all students (see Hattie, 2008; Hayes et al., 2006; Iivari et 
al., 2020; OECD, 2005, 2011). In the digital age, as we have been arguing in this 
chapter, there is a clear need for rethinking teachers’ professional knowledges and 
practices, given the affordances of the new technologies. These new knowledges and 
skills include: technical proficiency in the use of hardware and software; pedagogical 
know-how in the design and implementation of teaching sequences, using these 
digital materials; deep knowledge of the curriculum and evaluation practices; acute 
evaluative capacities; abilities to discern the value of digital technologies and data as 
fit-for-purpose; and capacities to collect, interpret, and discern the meaning of data 
so as to inform professional practices. 

On this point, Cope and Kalantzis (2016) comment astutely, “to teach and 
learn in such environments requires new professional and pedagogical sensibilities. 
Everyone becomes to some extent a data analyst” (p. 8). Data have become cri- 
tical to teachers’ daily work and a significant focus of and complement to teach- 
ing (Hardy & Lewis, 2017; Lewis & Holloway, 2019; Takayama & Lingard, 
2019). Importantly, educational data can come from multiple sources, including 
the digital. While the very definition of data literacy is currently undergoing 
transformation, given these new and rapidly emerging digital contexts, teachers 
are well aware of the need for clarity about the characteristics of quality digital 
resourcing. Pangrazio and Sefton-Green (2019) note that data literacy “involves 
both critical understandings of the technological infrastructure and the political 
economy of digital platforms, as well as strategies and tactics to manage and pro- 
tect privacy and resist being profiled and tracked” (p. 7). This draws us to the 
nexus between data literacy and digital literacy and the new capacities for using 
both to promote learner growth and track performance in meaningful and pro- 
ductive ways. The risk is that the easily measured can come to dominate what 
counts as valued performances in learning. This is where human judgement 
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remains so important. Applying Zuboff (2019) to schooling, there are two ques- 
tions here: “Can all the important elements of classroom practices and student 
learning be datafied?” and the additional normative one, “Should all such 
experiences be datafied?” 

Building capacity in effective educational data usage involves the under- 
standing of multiple sources, purposes, and aims of data. Skills involve analyzing 
and turning data into instructional actions, beliefs, sensemaking, and working 
collaboratively with data in school teams (Ikemoto & Marsh, 2007; Van Gasse 
et al., 2016). This is the context in which data dashboards provide a way to 
catalogue data for access and use by teachers and school leaders, seeking infor- 
mation about how well their schools are tracking against themselves over time 
and against other like schools (see also Gorur & Arnold, Chapter 10). The 
dashboards provide the means for making the data accessible and useable. 
However, they bring with them the demand for new forms of teacher expertise, 
what is usually included in the concept of data literacy. As teachers and school 
leaders work with digital data, a key aspect of professionalism is how to trans- 
form such data into action in the classroom and at the whole school level. 
Teacher expertise is necessarily processing data so that the data then become 
information and evidence for considering appropriate pedagogical practices. 
Here we see the important distinction between data and information, and data 
and evidence, with information and evidence referencing the processing and 
evaluation of data. Increasingly, systems and schools are looking to use digital 
architecture in productive ways for informing policy and practice and for 
monitoring and reporting on student learning. 

Systemic data can trigger tensions between system and school site validity in 
various types of academic assessment. This tends to occur when priority is given 
to systemic data as the source of truth and typically works with standardization, in 
contrast to site validity, which works with the idiosyncratic specificities of stu- 
dents in classrooms (Freebody & Wyatt-Smith, 2004). Bourdieu (1998) makes a 
related point that the logics of practice of the state work with the universal and 
are manifested in public policy, which is then applied for example universally in 
schooling systems to the this-ness of individual schools, thus always ensuring some 
infidelity in the enactment of policy in different schools situated in variable con- 
texts. Related, Thomson (2020) talks about the separation in schooling systems 
between “systemic data related to equity and effectiveness” and “school-based 
evaluations necessary for diagnostic and improvement work” (p. 216). This ten- 
sion between system and site validity (Freebody & Wyatt-Smith, 2004) raises the 
question of who should control the field of judgement in schooling (Ball, 2003). 
It also raises the contemporary issue of the impact of the emerging shift to 
increased control in the digital space of psychometricians, test constructors, data 
and learning analysts, and policymakers (Nichols & Berliner, 2007; Popham, 
2014). It is in that context that we stress the significance of teacher judgement 
and their evaluative expertise. 
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Chapters in the Collection 


Ben Williamson in Chapter 2 uses science and technology studies to provide a 
partial genealogy of developments in education data science, specifically the emer- 
gence of the learning engineer from the earlier construction of the education data 
scientist. The focus is on what Williamson refers to as the “big data imaginary,” 
how this is materialized in specific infrastructural arrangements and embodied in the 
practices of data scientists. In his analysis, Williamson problematizes the current 
fixation on big data in education, as learning sciences and engineering develop- 
ments enable “automated knowledge discovery.” He demonstrates how the con- 
cept of the learning engineer is based on a mix of cognitive science, psychology, 
and neuroscience and involves a move from organizational data-based analytics to 
individual learner analytics that open up to data analysis the internal substrates of 
learning (emotions, brain activity/reactions, eye/iris movements, etc.). The heavy 
involvement of EdTech companies such as Pearson, and philanthropists, such as the 
Chan Zuckerberg Institute and the Bill and Melinda Gates Foundation, in these 
developments is also demonstrated. Williamson’s genealogy provides stunning 
exemplification of digital disruption in education and its potential ramifications for 
the work of teachers, teacher professional judgement, student learning and school 
experiences, and the constitution of democratic public schooling systems. We note 
how Williamson has demonstrated the ways external developments outside of the 
education policy community are affecting the work of schools and teachers. 

Erica Southgate, in Chapter 3, seeks to translate and interpret the technical, 
mathematical, statistical, ethical, and legal/regulatory issues associated with artifi- 
cial intelligence (AI) and machine learning (ML) for teachers and education pol- 
icymakers. This is necessary knowledge so that both groups can have informed 
professional conversations about AI and machine learning. Such conversations are 
necessary so that professional teachers are not merely the objects of EdTech and 
system developed AI based reforms. The main focus is on the ethico-governance 
matters raised by these developments, produced by both large and small EdTech 
companies, and their take-up in education systems and schools. Clear explana- 
tions are provided of different types of AI and related ML and their usage in 
schools — Artificial Intelligence in Education (AIEd), while noting the often- 
anthropomorphic construction of AI and ML and exaggerated expectations cur- 
rently about what they can or might be able to do. The chapter concludes by 
considering the ethico-governance issues that these matters precipitate. A number 
of issues are raised for teachers, schools, and systems in respect of AI in education 
in relation to its design, use, and governance. These are: the need for necessary 
foundational knowledge for teachers and policymakers, explainability of AI, fair- 
ness, transparency (difficult given the at times opaque character of algorithmic 
decision-making that underpins both AI and ML), and accountability, while 
recognizing that AI and ML developments have outpaced contemporary gov- 
ernance structures and regulatory frameworks. 
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In Chapter 4, Brian Lee-Archer begins with what he refers to as a challenging 
paradox: societies’ increasing use of digital data to help address serious problems 
facing the planet, and citizens’ growing lack of trust in their governments and public 
institutions. Within this frame, the chapter focuses on the emerging relationships 
between humans and machines (often referred to as “human+machines’”), in a phase 
of human history increasingly dominated by artificial intelligence (AI) and the use of 
big data (Daugherty & Wilson, 2018). Lee-Archer explores how the use of digital 
data is a significant contributing factor in declining trust and presents the strong case 
that the way humans work with machines is vital to arresting this decline, and 
more positively, building public trust. He argues that stronger societies and better 
educational outcomes for children are dependent, in part, on the human element 
in decision-making. He proposes that stronger societies use existing assets 
including data, and promote citizen-led initiatives, to achieve the necessary bal- 
ance between managing societal risk and rewards. He argues that such balance 
can be achieved through innovation from within the human dimension, guided 
by a dynamic base of digital data. While Lee-Archer recognizes that trust in 
public institutions and government is constantly challenged, a counter-balancing 
move would be for policymakers to adopt what he refers to as a structured pro- 
cess driven approach to AI and big data to provide a basis for building public 
confidence in digital initiatives. Finally, Lee-Archer asserts that a key issue in this 
approach is how policy promotes citizen-led initiatives so that people control 
their affairs in a democratic and inclusive way. 

In Chapter 5, Kalervo Gulson, Andrew Murphie, and Kevin Witzenberger use 
the idea of Amazon Go as an intellectual springboard for education and open 
wide “the intensifying and disruptive capacities of AI in education.” The authors 
put to readers the unsettling idea that: “for the first time in history we have no 
idea what to teach in schools.” They propose the use of speculative approaches as 
ways of thinking through the implications of the future, as it involves artificial 
intelligence (AI), disruption, and intensification. They imagine the implications of 
a kind of Amazon Go for education. The chapter presents scenarios of changed 
roles of schools, universities, teachers, and forms of organizations that develop as 
AI is introduced. The authors have based the chapter on secondary evidence and 
acknowledge inherent limitations in how we can predict and plan for the future 
of education and schooling. They challenge readers to focus on what they refer to 
as “preparedness in relation to the unexpected.” They locate the chapter within 
the conceptual resources from critical technology studies in education, new media 
studies, and science and technology studies. The chapter is presented in four parts. 
They first outline what they refer to as their “conceptual apparatus and metho- 
dological concerns,” based on a future yet to be born, a future that invites spec- 
ulation and the challenge for educational research of examining conditions which 
are yet to appear. The second part of the chapter provides a review of literature, 
and speculations about the intensification and disruptive possibilities of AI in 
education. With the continuing focus on AI, the authors posit that the current 
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organization of education may shape the ways new AI technologies are adopted. 
This is a future where the influence of corporate interests and technological 
advances also shape the adoption of AI and, in turn, the nature and function of 
education. They then turn to consider possible substantive disruptions to educa- 
tion, addressing issues of provision and knowledge. They conclude the chapter by 
attempting to provide a sense of the issues or questions that they think require 
further examination. Drawing on Ross (2017), the authors recognize the limita- 
tions of speculative methods and also highlight that digital education research 
currently lacks what they refer to as “an imaginative resource to take a strong 
position at the edges of educational change.” 

Anna Hogan and Sam Sellar in Chapter 6 investigate the impact of the for- 
profit edu-business Pearson and the company’s subsequent role in public educa- 
tion as they extend their corporate strategy. Through a review and discussion of 
the digital transformation of the Global Education Industry (GEI) and followed 
by a scoping review of Pearson’s commissioned reports in education, Hogan and 
Sellar critically analyze Pearson’s priorities and broader corporate strategy. Key 
points include the digital transformation of teaching, curriculum, and assessment, 
and the disruption of schools and schooling. Pearson’s influence on the role of 
the teaching profession is a key concern, in addition to the company’s collection 
of digital data and the subsequent issues that surround privacy, ownership, trans- 
parency, bias, and openness. Of particular note, Hogan and Sellar argue that 
consent is not always explicitly sought, and as a result, this complicates questions 
around responsibility and data ownership. These are significant concerns in regard 
to the impacts and the place of such private for-profit edu-business, particularly in 
poorer nations. There are further risks of deprofessionalization of teachers, as the 
teacher is reimagined as a “facilitator,” and the consequent transformation and 
disruption of public education. 

Laura Engel and David Rutkowski in Chapter 7 focus on the costs of partici- 
pation in ILSAs, specifically the costs of the US’s participation in both PISA and 
TIMSS. Here they build on their earlier work (Engel & Rutkowski, 2020). They 
argue that this focus on costs is important in terms of the educational benefits or 
otherwise of such participation and given the reality that ILSAs both drive the 
datafication of educational governance and are an effect of it. The costs fall into 
two categories: the overhead costs paid to the OECD and IFA for participation, 
and the costs of implementation of the tests inside the US, including the direct 
and indirect costs in each area. Using publicly available data, some of which is 
quite difficult to access, they document the amounts expended, which are very 
substantial. In various cycles of both PISA (every three years) and TIMSS (every 
four years), some states (sub-national units) have participated as individual entities 
in addition to the national sample. There are further state-level costs involved 
here. Engel and Rutkowski then consider potential costs/benefits of participation 
in these two ILSAs. In so doing, they review the cost/benefit methodology and 
point out the futility of trying to put a monetary amount on the benefits, which 
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are both direct and indirect. In considering costs/benefit analysis in respect of 
ILSAs, they point out the lack of clarity of the US’s rationale for participating in 
them, which limits accountability. They recommend broader educational com- 
munity involvement in the establishment of this rationale and call for more 
research on evaluating the benefits of participating in ILSAs in the USA and in 
other nations. 

Sigrid Hartong, Annina Forschler, and Vito Dabisch in Chapter 8 investigate 
data infrastructures and the (ambivalent) effects of increasing data interoperability 
in German schooling. This case study explores the very different political contexts 
for digital datafication and disruption of education in Germany, against the 
backdrop of German federalism. They note that although Germany, compared 
with other nations, has been a latecomer to datafication and digitalization in 
education, there has been a transformation and expansion of centralized perfor- 
mance data in school systems, and since 2012, a growth of digital learning tools. 
Yet, this has also been met with much greater caution in local sites of decision- 
making and implementation. They explore the emergence of datafication in 
German schooling via the transformation of standardized testing in state school 
monitoring and provide examples of school platforms and learning management 
systems. This chapter demonstrates the significance of the specificities of national 
contexts to understanding emerging datafication in and across national school 
systems. This is the broad point that digital disruption plays out in path-depen- 
dent ways in different nations and in different schooling systems. 

In Chapter 9, Jessica Holloway and Steven Lewis use the concepts of “datafi- 
cation” and “surveillance capitalism” in the context of schooling in Texas, USA. 
They lift the lid on a new type of “black boxing” in education data and assess- 
ment, revealing how schooling is being fundamentally reconceptualized by a mix 
of new partnerships, digital data, new technologies, technicians, and statisticians. 
They identify the influence of public-private partnerships involving Texas public 
schools and three organizations: SAS Institute Inc., Responsive Learning, and the 
National Institute for Excellence in Teaching (NIET). They discuss how these 
groups have worked in collaboration with the Texas Education Agency to build 
the Texas Teacher Evaluation Support System (T-TESS) as the new data that 
serve to manage schooling processes. Holloway and Lewis reveal how value is 
ascribed to the work teachers do — teacher effectiveness — and more fundamen- 
tally, how teacher identity is being changed and reconstituted as data. More than 
this, the authors shine the spotlight on how new actors in schooling with exper- 
tise in digital data, technicians, computational processing, prediction, and statis- 
tical calculation are contributing to partnerships with schools in Texas, changing 
how benefit is understood in relation to such partnerships. In opening up the 
black box of datafication in this case, they present for readers a stark picture of 
data in a mix of private, business-oriented purposes in education, with impacts on 
teacher professionalism including professional judgement. This is a cautionary 
tale! 
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In Chapter 10, Radhika Gorur and Ben Arnold explore the sociotechnical 
imaginary of “governance by dashboard” in education. The authors argue that 
although the concept of digital data dashboards is seemingly transparent, these 
dashboards also perform their own invisibilities. As such, there are concerns for 
the goals of accessibility, reform, governance, quality, and equity in education 
provision. Empirically, Gorur and Arnold focus on the Global Education Policy 
Dashboard (GEPD), currently in development by the World Bank in conjunction 
with DfID (the United Kingdom’s Department for International Development, 
now being restructured) and the Bill and Melinda Gates Foundation. Through an 
analysis of various documentation, themes explored in this chapter include “The 
Imagined System,” “The Imagined Classroom,” “The Imagined User,” and dis- 
cussion about the contradictions, challenges, and the various complexities of such 
a sociotechnical imaginary of education. This is significant, as the presentation, 
visualization, and curation of data via digital data dashboards can affect the ways 
in which problems are understood, and the ways in which policymakers from 
low-income nations are viewed — as unsophisticated and requiring external 
expertise. Importantly, as the authors note, this is also concerning particularly for 
nations in the Global South, where there is the risk of neo-colonial policy 
directions from the Global North and from edu-businesses located there. 

In Chapter 11, Greg Thompson examines the evolving nature of digital 
assessment or what he refers to as “next generation assessment.” He notes that 
standardized testing has been problematic as it is often linked to a top-down form 
of accountability. To investigate “next generation assessment,” he delves into the 
Australian Online Formative Assessment Initiative (OFAI), with a focus on 
learning progressions. A key concern in this regard is the place of teachers in this 
context and their potential displacement in such automated and algorithmic sys- 
tems. To frame his chapter, Thompson first examines the educational and policy 
contexts in Australia, followed by consideration of the implementation of 
NAPLAN, and the emerging debates around digital technology in this assessment 
space, before discussing concerns about the OFAI. As a result, he explores a 
solution to adopt a technical and more democratic focus, or a “technical 
democracy” (drawing on Callon et al., 2009; Thompson et al., 2019), which 
considers the roles of decision makers and stakeholders in these emerging assess- 
ment systems. 

The suite of Provocations, collectively structured as Chapter 12, deals with the 
very real impact of the COVID-19 pandemic on digital disruption in education. 
We decided these provocations were needed as recognition that the social, which 
is the focus of our research, continues to change. In the pandemic crisis, some 
problems are made more visible, which are usually hidden from view and from 
political consideration. The pandemic has brought to the fore, for example, 
matters of inequality in relation to access and participation in the digital revolu- 
tion (e.g., the digital divide) and in the provision of schooling and related 
opportunities for learning. With the move to so much online schooling from 
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home, the reality of the digital divide has become very apparent. Many young 
people from poor families do not have laptops or access to the Internet. The 
pandemic has raised the issue of whether these matters should be universally 
provided by the state. A positive move in some parts of Australia has seen the 
state step in to ensure everyone has laptops and connectivity. The pandemic has 
also witnessed the EdTech industry use the crisis to intervene further in many 
aspects of school provision, with some suggesting schooling at home during the 
pandemic, using all of the new technologies, has offered a glimpse of the future, a 
sort of online driven deschooled society (Illich, 1973). The pandemic has fore- 
grounded these matters and potentially increased the speed and extent of digital 
disruption in education. It is this set of matters that these provocations so effec- 
tively and concerningly address. 

Neil Selwyn, author of What is digital sociology? (2019) and Should robots replace 
teachers? (2019), argues persuasively that COVID-19 means that we need to see 
the digitalization of education through the lens of the pandemic. He makes two 
important points: 1) COVID-19 is being used as a cover for accelerating an 
EdTech framed future of schooling, the dismantling of established schooling (cf 
Hursh, 2017), and 2) in that context, there is a pressing need for critical education 
scholars to articulate a counter and alternative policy narrative and imagined 
future for socially just public schooling. One purpose of the Digital disruption in 
teaching and testing collection is to form a basis for such counter narratives. 

Nick Couldry, co-author with Ulises Mejias of The costs of connection (2019), 
which documents the ways data is colonizing human life, makes two significant 
points about the impact of the pandemic on education in his provocation. The first 
relates to the possibilities of more intense governing of teachers and students as 
school systems move to online teaching and learning. The second is the danger of 
substituting the deep inequalities of different home lives of students for the 
“common resource” of schools and classrooms. The latter, of course, still remains 
an aspiration, given inequalities between schools and in relation to funding, but it is 
at least an aspiration and focus of political mobilization and policy interventions. It 
is the former that Couldry focuses on, while noting that the present moment of 
home schooling is in some ways akin to Ivan Illich’s (1973) “deschooled society,” 
while pointing out that Illich always saw potential dangers in such deschooling in 
ushering in other axes of power. As with Selwyn, Couldry suggests that what we 
are seeing is the acceleration of trends already underway and the use of this crisis as 
an opportunity for edu-businesses to intervene further in schooling and classrooms. 
Specifically, he evocatively suggests that the management of the classroom by the 
eye and voice of the teacher will be replaced by surveillance via online resources, 
continuous assessment practices, and data collection. 

Anna Hogan and Ben Williamson, drawing on their report for Education 
International, Commercialization and privatization in/of education in the context of 
COVID-19 (2020), suggest that the pandemic has enabled the EdTech companies 
to characterize themselves as emergency responders as 1.6 billion children moved 
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to remote online learning across the globe. Utilizing Naomi Klein’s (2007) con- 
cept of “disaster capitalism,” they argue that the EdTechs will use this opportu- 
nity for further privatization of schooling to their long-term benefit and profit. 
They document the global network of policy actors (international organizations, 
philanthropies, and EdTech companies) driving these changes and also note the 
significance of private infrastructures that manage online learning. 

Sotiria Grek in her provocation documents another continuity pre- and post- 
pandemic, namely the distrust of teachers in policy terms, particularly in England. 
She documents how the health pandemic has produced a new epidemic of “lin- 
guistic signifiers,” which represent quantitative measures of both everyday life and 
politics; think of the infamous R measure of the potential impact of COVID-19 
infection rates. She suggests, rightly in our view, that the pandemic has also enhanced 
the standing of science. It is, however, the new moral discourses that have arisen in 
the context of the pandemic that she is concerned with. She shows how key or 
essential workers have become heroes, yet in England teachers as front-line workers, 
have been constituted in government discourses as anti-heroes, not worthy of trust. 
What is interesting in Grek’s provocation is this long-term demonizing of teachers. 
What we see is the path dependent ways the pandemic plays out discursively in dif- 
ferent national contexts. In Australia, in sharp contrast, teachers have been included 
in the category of essential workers and it seems that public recognition of their work 
and professionalism has been enhanced during remote schooling as a result of the 
pandemic. The situation in England, Grek notes, has also provided a further business 
opportunity for the EdTech companies. 

Sam Sellar, in his provocation, draws on the work of Zuboff (2019) on “sur- 
veillance capitalism” to suggest three strategies that might enable us to resist its 
forages into schooling and its further privatization. He notes how the pandemic 
has opened a space for EdTech companies seeking to profit from the temporary 
shutdown of schools and the move to online learning from home. Using Zuboff, 
he talks of the ways in which, under surveillance capitalism, all human experience 
is being rapidly datafied as we use multiple digital online platforms in our lives. 
This datafication is then used as a commodity to profit from. This rendering is 
accompanied by a surrendering of our privacy and experiences to tech companies, 
as Zuboff insightfully puts it. Yet, it is strategies of opposition to surveillance 
capitalism and its playing out in schooling that Sellar is concerned with and in the 
context of the pandemic. 

The three strategies of resistance Zuboff developed are taming, hiding, and 
indignation. The first works in relation to demands for legal and legislative pro- 
tections of our data privacy — a difficult to achieve outcome given the power and 
influence of the huge tech companies. The second strategy is hiding, whereby we 
mask our identities and activities from digital platforms. However, as Sellar 
argues, such a strategy leaves surveillance capitalism intact. The final strategy is the 
one he holds most hope for: indignation. As Zuboff argues, the current situation 
of surveillance capitalism “teaches us how we do not want to live.” It is in such 
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indignation that Sellar argues there are political possibilities for repairing the 
infrastructure of public schooling and recognizing the importance of the collective 
experiences there, as opposed to the individualization that accompanies online 
modes of curriculum, pedagogy, and assessment. Sellar argues that the experiences of 
home schooling and online learning during the pandemic have driven home the 
significance of schooling as a centrally important social infrastructure. He asserts, 
rightly in our view, that schools are very important “public things.” 


Conclusion 


Change is a characteristic of the human condition and human societies. The 
current moment of digital disruption provides striking evidence that confirms this 
observation, demonstrating accelerated change in relation to the formation and 
functioning of societies and human identities. Digital technology and computa- 
tional power and capacity have advanced exponentially since the 1960s. Compare 
the computational capacity and processing speed of the smart phone today with 
those of the mainframe computers of the 1960s and 1970s. The impact of digital 
disruption is often referred to as ushering in a Fourth Industrial Revolution. This 
references inter alia the ways the digital revolution has reshaped production and 
consumption practices in the economy, and the nature and provision of 
employment opportunities and the skills needed for work and civic contributions. 
More than this, however, the digital revolution is having an impact on the 
broader social fabric and human identities. We acknowledge, however, this is 
manifested differently in the nations of the Global North and the Global South, 
with the potential for the digital divide to exacerbate inequalities that have been 
growing over past decades (Piketty, 2014) and also in relation to schooling 
(Chmielewski, 2019). 

The digital has enabled new opportunities for human connections, locally, 
nationally, and globally and at speed. It has given ready access to knowledge in 
unprecedented ways and has also enabled knowledge generation to solve a whole 
range of human problems, across health, well-being, business, and tracking of 
economies and finance systems. We have now at our fingertips opportunities to 
follow and track (surveil) individuals, groups, and wider populations. Education 
has been slower in relation to other domains in taking up the digital. 

The digital revolution has occurred in a changing context of a post-Cold War 
world that saw the emergence of a global economy, the dominance of neo-liberal 
ideology, the rejection of Keynesian welfare states, and restructured state 
bureaucracies, first through new public management and subsequently through 
network governance. This context has seen the digital revolution, driven by the 
interests of huge tech companies with profit motives, assert an unprecedented 
authority over governance and decision-making at system, and in turn, local 
levels as nations have moved to digital governance (see Grek et al., 2021), 
including digital governance of education (Williamson, 2017). 
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Added to this mix is the potency of big data, which have become central to 
public policymaking and government steering of the social that has become 
increasingly quantified and datafied (Mau, 2019; Zuboff, 2019). In education, it is 
widely reported that governments have an insatiable appetite for data. These data 
are now constituting and remaking education systems (Lawn, 2013), through 
accountability systems of a range of kinds used to track the performance of 
schools, teachers, and students. These performance-based modes of accountability 
hold schools and teachers to account. There has been heavy edu-business and 
EdTech involvement in these developments, as many chapters in this collection 
document. The transparency of these, and the ways in which they are remaking 
the social fabric of schooling, however, are less apparent. It was the big EdTech 
companies that created the standards for data infrastructures in education. The 
schooling of young people at home during COVID-19 has revealed a range of 
benefits of the digital in education, but also a range of shortcomings. These are 
well documented in the provocations in the collection. 

In this chapter, we have documented the affordances of technology in education, 
noting that the social remains in a state of constant flux. We have adumbrated how 
the digital revolution is moving to provide big data at global and national levels, and 
education is potentially poised to use this for policymaking and educative purposes. 
Education data are valuable assets, a phenomenon already well recognized by the 
EdTech companies. 

The digital revolution has already had some impact on teachers and teaching. 
We would stress here again that teaching is a relational and cultural activity. In so 
recognizing, we are aware that the digital revolution might continue without 
recourse to the profession, and in doing so, accelerate rapid deprofessionalization. 
In that concerning context, we argue the need for teachers’ professional knowl- 
edge and judgement to be utilized in conjunction with the affordances of the 
digital. This will demand new collaborations and networks, along with new 
knowledges and expertise in digital and data literacies. As we argued earlier, the 
integrating of expertise across fields and the collaboration of individuals and 
groups with this expertise are critical in optimizing human+machine interactions 
for better futures. This is much more than “technical know-how.” 

We have also documented positives that might arise from the conjoining of 
digital change and new approaches in assessment practices. There are real oppor- 
tunities, for example, for new designs in assessment eliciting a wider range of 
knowledge and skills, and greater flexibility for “assessment to be less time critical 
and location specific.” There are also opportunities for addressing the desirable 
authenticity of assessment, enabling young people to work collaboratively in 
teams, focusing on creativity and problem-solving. Digital technologies “can 
make it possible to vary the location, timing, and length of assessments to suit the 
individual needs and preferences” (Timmis et al., 2016, p. 7). 

In the context of this rapid change, we present some value propositions. First and 
foremost, we argue that humans should knowingly drive the digital revolution 
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broadly and in education specifically. Put succinctly, the institutional provision of 
education is central to the production and shape of human futures. We use “know- 
ingly” here to refer to the social justice position that the affordances of the digital 
revolution be utilized to benefit all and be geared towards strengthening educa- 
tion as and for the common good (UNESCO, 2020) to focus on the place of the 
human. Schooling is central to the socially just provision of opportunities for all, 
irrespective of social class background, race, ethnicity, gender, or ability. The 
affordances of the digital and data must be used to enable these aspirations to 
become reality. We note here Piketty’s (2014) telling observation that, “refusing 
to deal with numbers rarely serves the interests of the least well-off’ (p. 577). 
Here, drawing on Mau (2019), we make the cautionary observation that the 
datafication and quantification of the social might very well reinforce a compe- 
titive individualism where “people no longer fight collectively” for “a fair dis- 
tribution of wealth” (Mau, 2019, p. 169), but simply pursue their own interests, 
reinforcing a central premise of neoliberalism. Further, Mau characterizes that 
“the metric society is a mass of individuals in competition with each other, not a 
solidary or cooperative community” (p. 169). 

Second is the proposition that teacher professional judgement and evaluative 
expertise are core in decision-making for realizing the potential of the digital in 
schooling. It is here we note the reported “remarkable innovation in the responses of 
education in the COVID-19 crisis, with those systems most engaged with families 
and communities showing the most resilience” (UNESCO, 2020, p. 5). This 
observation points to the critical need for new partnerships and collaborations in 
advancing education. It also acknowledges the centrality of professional teachers and 
quality teaching to the provision of quality schooling for all. 

Third, we propose that a new knowingness is necessary across social and policy 
actors in relation to the utilization of the data, the use of algorithms and the 
uptake of the affordances of virtual reality in curriculum, pedagogy, and assess- 
ment. There needs to be a real mindfulness that, as the digital change plays out in 
education, the gap between the “haves and have nots” does not broaden (stu- 
dents, schools, systems, and sectors), both within and across nations. Here we 
return to our concept of “socio-technical education data imaginary,” highlighting 
appropriate human+digital and human+data relationships (subsets of human 
+machine considerations) for determining the appropriate place of the digital and 
of data, for shaping possible futures of education systems, schools, and young 
people’s learning. 

We well recognize the lure and the attractiveness of the customization and 
personalization of learning for growth as promoted by digital interventions. 
However, positive connotations here can mask the underlying standardization 
necessary to deliver such personalization. There is a deep irony in the inter- 
weaving of personalization and standardization: the former offers the promise of 
individually designed learning pathways and progressions, whereas the latter 
operates on the assumption that individual performance is to be tracked using a 
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predetermined underpinning scale. We stress here with digital assessments that 
personalization and standardization are inextricably conjoined. We would also 
point out that edu-businesses desire standardization of testing and curriculum 
materials and reporting to broaden their market opportunities and make products 
at scale (Verger et al., 2017). 

Countering the lure are other risks in digital and data disruption, including the 
digital rights of the child (see Livingstone et al., 2019) and also those of parents 
and teachers. There have been some moves to counteract this by organizations 
like UNICEF. As we have noted, digital disruption can bring both benefits and 
disadvantages. Regarding the critical issue of children’s rights, Livingstone iden- 
tifies how digital environments facilitate, 


the rapid spread and extensive networking of information and communica- 
tion in ways that can be aggregated and analyzed on a global scale, which can 
be both beneficial and harmful. Today, all our digital interactions generate 
data that can be shared, searched, altered or exploited by third parties, and 
the consequences may be exciting or unwanted, and are often unintended 
and unpredictable. 

(Livingstone, 2019, para 3) 


More broadly, there are important questions about the privacy and security of the 
massive amounts of data collected and stored by both governments and private 
sector actors. These concerns apply to the collection and storage of data in edu- 
cation and the further risks of hacking. Added to this, are numerous questions 
about data ownership and transfer. These questions are evident in concerns 
globally, about data security, invasion of privacy, and potential on-selling of data 
to third parties for profit. Governments around the world and inter- and supra- 
national organizations (e.g., the UN, UNESCO, and EU) are undertaking the 
critical work of developing appropriate legislative frameworks to deal with these 
complex and rapidly evolving matters. The technology, however, has already run 
well ahead of these developments. Most of humankind is in a consequential game 
of catch-up. 

This book provides research-informed accounts of the multiple ways digital 
disruption and data use are beginning to profoundly reshape education systems, 
policymaking practices, and the work of schools and teachers. This reshaping 
comes with implications for young people’s learning, identities, and their experi- 
ences of schooling with significance for their futures. Such analyses as provided 
are necessary to begin constructive dialogues across knowledge domains and 
groups with legitimate interests in education policy and the future of schooling. 
These include students, teachers, teacher unions, parents/carers, the wider com- 
munity, school leaders, system leaders, policymakers, and researchers. What we 
have attempted in this introductory chapter is to set the scene to the chapters 
included as a way of opening up these necessary conversations about educative 


Transforming Schooling 37 


and democratic uses of data and the digital for policymaking and how these might 
amplify and augment the new teacher professionalism requiring new knowledges 
and expanded evaluative expertise. 

Beyond this, we invite readers to think about how this kind of reshaping of 
schooling links to desirable imagined futures, where considerations of the 
common good are openly debated and transparently frame policymaking and 
professional practices in education. How can we re-imagine schooling for a more 
democratic and socially just future (Apple, 2013)? It is now imperative to go 
beyond the lure of the efficiencies and effectiveness of technologies and turn to 
how they should serve humankind and its advancement. In education, these 
matters clearly play out in data usage, policymaking, and new forms of digitalized 
testing and assessment. It is already abundantly clear that education is a new 
marketplace; data are valuable commodities and assessment remains among the 
highest of education policy priorities. In that limiting social imaginary, there is 
also a very real prospect of teachers and students becoming datafied objects of 
policymakers and EdTech companies. There is the added danger of such changes 
being imperceptible. However, this is not inevitable and is why we offer our 
alternative normative concept of a “socio-technical education data imaginary.” 
While many questions remain, of one thing we are sure: we are at the very 
beginning of addressing profound educative, ethical, legal, privacy, and technical 
complexities, along with aporias of human+data interactions in general, in edu- 
cation, and machine learning in particular. We offer this collection as a step 
towards opening up these matters as part of the necessary ongoing conversations 
of humanity. 


Notes 


1 In writing this chapter, we have drawn on our earlier Education Research and Foresight 
Working Paper: Wyatt-Smith et al. (2019). Digital learning assessments and big data: 
Implications for teacher professionalism. Education Research and Foresight Working Paper 
25. UNESCO. 

2 The docudrama “The Social Dilemma” (Coome et al., 2020) deals with these matters, 
as does David Eggers’ (2014) novel and related film, The Circle. 

3 In this chapter, we have used “black boxing” to refer to a number of different processes 
in governance, assessment practices, and teacher judgement and decision making. 

4 The term validity is taken to refer to what is assessed and the extent to which the 
assessment or test is measuring what it claims to measure. 

5 The term reliability is taken to refer to rater- and inter-rater consistency or agreement. 

6 ACARA does not endorse any product that uses its material or make any representa- 
tions as to the quality of such products. 

7 ACARA does not endorse any product that uses its material or make any representa- 
tions as to the quality of such products. 

8 See Manolev et al. (2019), for an overview, analysis, and concerns of Class Dojo and its 
potential for surveillance and performative classroom culture. A parental Class Dojo app 
has been developed to complement the data collected at school. 

9 See India Education Diary Bureau Admin. (2019). 
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AUTOMATED KNOWLEDGE 
DISCOVERY 


Tracing the Frontiers, Infrastructures, and 
Practices of Education and Data Science 


Ben Williamson 


Introduction 


The emergence of “big data” and “artificial intelligence” (AI) in education are clo- 
sely associated with developments in assessment and testing, such as learning analy- 
tics, adaptive tests, and personalized learning platforms. Less visibly, however, big 
data and AI practices are now being used as the basis for new scientific approaches to 
the understanding of learning and educational processes. Data-intensive forms of 
scientific discovery and knowledge production are being promoted extensively by 


” 


“education data scientists,” “learning scientists,” and “learning engineers,” with sup- 
port from influential institutions including the Organization for Economic Coop- 
eration and Development (OECD) and the Chan Zuckerberg Initiative. Drawing 
extensively on technical engineering and data analytics advances in knowledge dis- 
covery — from disciplines including psychology, cognitive science, neuroscience, and 
genomics — these developments exemplify the emergence of a new science of 
policy-relevant research and development in education. 

This chapter adopts the science and technology studies method of unpacking 
the expert practices and material apparatuses by which science makes its objects 
legible for intervention. It provides an analysis of “automated knowledge dis- 
covery” as an emerging epistemology and a set of expert data practices in edu- 
cational research, tracing the implications of applying learning science and 
engineering insights to the tasks of educational policy and practice. As such, this 
chapter represents a partial genealogy of the formation and evolution of education 
data science as a site of expert practice and knowledge production. As a method, 


genealogical analysis traces how contemporary practices and institutions 
emerged out of specific struggles, conflicts, alliances, and exercises of power... 


DOI: 10.4324/9781003045793-2 


46 Williamson 


its intent is to problematize the present by revealing the power relations upon 
which it depends and the contingent processes that have brought it into being. 
(Garland, 2014, p. 372) 


Genealogies seek to trace out the historical processes and conditions out of which 
contemporary practices emerged and upon which present-day practices depend. 
As Gehl (2015) has shown in his “partial genealogy of the data scientist,” multiple 
historical threads, conflicts, bifurcations, con- and dis-junctures have produced 
the expert practices, subjects, and apparatuses of data science. These include dis- 
ciplinary conflicts and encounters, technological developments, methodological 
advances, and in large part, sectoral encounters between academic research and 
commercial imperatives. 

Taking up this genealogical method, this chapter represents an attempt to 
problematize the current fixation on the use of “big data” in education, in gen- 
eral, and more specifically to interrogate how the subject that mines and analyses 
it — the education data scientist — has come to be. The focus is on three particular 
interrelated aspects of this genealogy of education data science. First is the “big 
data imaginary” that has come to animate much education research, policy, and 
practice. Secondly, this imaginary is argued to be materialized and instantiated in 
specific infrastructural arrangements of technologies, methods, and organizations, 
all framed by particular social, economic, and political concerns and contexts. 
Thirdly, the imaginary is embodied in the particular practices of education data 
scientists. The presentation of some of these genealogical threads problematizes 
the idea of education data science as an objective science of learning, and instead 
highlights the messy convergences and contingencies across sectors, institutions, 
and practices that have enabled data science to become authoritative in educa- 
tional knowledge production. The central argument of this chapter is that a series 
of genealogical combinations have enabled education data science to push back 
the boundaries of data-informed knowledge discovery beyond the exterior 
boundary limits of the human body, and to open up intimate interior functions 
and mechanisms of students’ bodies, affects, and brains to increasingly automated 
forms of data collection, analysis, and knowledge discovery. 


Data Science Frontiers 


Data science has been defined as “an umbrella term to describe the entire com- 
plex and multistep processes used to extract value from data” (Irizarry, 2020, para 
5, drawing on Wing 2019) and involves various practices of data generation, 
collection, organization, processing, analysis, and interpretation. As an academic 
discipline, data science first emerged in the 1960s as research scientists sought to 
accommodate technical practices of data analysis and computing within the wider 
field of statistics. As a profession, data science was popularized only in the 2000s 
as “a new type of project was becoming more and more common in the tech 
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industry: extracting value from messy, complex, and large datasets” (Irizarry, 
2020, para. 3). It is, however, not an entirely coherent field formed around a 
consistent or coordinated professional set of practices and epistemologies, but 
rather a complex ecosystem that includes computer science, statistics, engineering, 
information and library sciences, law and philosophy, mathematics, data visuali- 
zation, and social and behavioral sciences, as well as innumerable areas of appli- 
cation (Meng, 2019). Crucially, it has been enabled by significant advances in 
computational capacity and processing power. “Data science is characterized more 
by modern technical resources and opportunities than by a set of methods or 
even specific applications” (Garber, 2019, para. 11). 

Although data science is clearly associated with developments in computing, 
algorithms, and analysis over the last half century, it is also located in a longer gen- 
ealogy of the rise of statistical technologies and practices going back two hundred 
years (Bigo et al., 2019). Contemporary data science, especially in its industrialized 
and commercialized forms, remains animated by a particular realist epistemology that 
is rooted in empiricist 19th-century natural sciences and statistics. It maintains that 
data represent reality as they appear in the world, as if separate from the modes and 
apparatus of computation that brought the data into being (Kitchin, 2014). From the 
perspective of the humanities and social sciences, however, data are understood not 
simply as objective numbers or facts to be computed, but as the constructs of 
knowledge or data infrastructures that govern how they are created, managed, used, 
and interpreted (Edwards et al., 2013). Many practices of data science involve com- 
plex arrangements of computational processes such as data analytics, machine learn- 
ing, recurrent neural networks, cognitive computing, and deep learning, as the 
“datafication” of natural and social worlds is defined by both material infrastructures 
and expert practices of knowledge production. “Data are selected, collected, orga- 
nized, and generated by humans, using the knowledge infrastructures available to 
them at the time” (Borgman, 2019, para. 14). 

As the practices, epistemology, and profession of data science have expanded 
their authority, value, and legitimacy, a new data analytics industry has formed to 
spread analytic processes, and the technologies they rely upon, into an increasing 
range of sectors and institutions. Beer (2019) describes the data analytics industry 
as an amalgamation of imaginaries, infrastructures and practices, which together 
push “data frontiers” into “previously uncharted and new territories” (p. 19). 
Data frontiers are boundary lines where data-informed processes reach their 
limits. The data analytics industry is constantly trying to push these boundaries 
back by deploying persuasive visions of the power of data analytics. Beer states, 
“they are active in ushering in the expansion and intensification of data within 
organizational and social structures. These visions of data analytics persuade, enact 
and produce notions and ideals that are designed to force back those frontiers” 
(Beer, 2019, p. 19). Beer further asserts for understanding, “the expansion and 
intensification of data as an active component of the social world” necessitates a 
turn to look at “the work that is being done at these data frontiers” (p. 19). 
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For Beer (2019), the work done at data frontiers is characterized by the pro- 
duction of imaginaries, the construction of infrastructures, and the enactment of 
practices. The concept of data imaginaries captures the “visions” projected by the 
data analytics industry, “the promises, futures, potentials and possibilities that are 
associated with data” (p. 9) and its embedded rationalizing discourses that shape 
the integration of data analytics into different settings. These imaginaries pave the 
way, Beer argues, for the engineering or expansion of data infrastructures that 
consist of software packages, data processing algorithms, AI techniques, and 
human operatives, which are required for analysis to take place. Data practices 
refer to the various sorts of labor involved in data analysis. These are practices that 
are embodied by figures, such as data engineers and data analysts, but are per- 
formed through hybrid configurations of humans and computers to discover 
knowledge from the identification of patterns in databases. With the development 
of “smart” analytics powered by machine learning, such data practices are 
becoming increasingly automated in “thinking infrastructures” (Bowker et al., 
2019). Data analysts are required to work within automated structures, while also 
constructing and managing them, ultimately “to mesh with machine learning and 
automated systems” (Beer, 2019, p. 101) in the process of automated knowledge 
discovery. 

Within education, the boundaries of datafication are also being pushed back by 
the burgeoning field of education data science. Perrotta and Selwyn (2019) sug- 
gest that automated knowledge discovery in education data science is not merely 
a technical or methodological accomplishment, but emerges from a complex set 
of relations and tensions between competing definitions of “learning,” dis- 
ciplinary perspectives and practices, models and datasets, technical platforms and 
infrastructures, academic and commercial entities, and epistemic cultures and 
economic interests. In this chapter, the focus is on two ways in which work is 
being done at data frontiers and on the genealogical threads that enabled such 
practices in the present. 

First, data analytics are being envisioned and put to work as powerful tech- 
nologies for knowledge discovery in educational research. Education data science 
has increasingly sought new ways of generating, collecting, analyzing, and inter- 
preting evidence of learning, and has proceeded from relatively limited and static 
datasets available from large-scale tests and assessments to “real-time” digital traces 
available from digital learning platforms. As such, a data frontier is being pushed 
back into processes and practices of knowledge production in the field of educa- 
tion, which is being accomplished through the construction and management of 
new analytic infrastructures and the development of new human-machine 
practices. 

Second, education data science is itself seeking to push a new data frontier further 
into the cognitive, psychological, neurological, and even biological mechanisms that 
underpin human learning. In some cases, and as this chapter illustrates, education data 
science has even begun to deploy sensor-based technologies, facial recognition, 
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neurotechnologies, wearable devices, or bioinformatics infrastructures to generate 
data from learners’ bodies and physical movements. The data frontier of education 
data science has begun, its advocates claim, to penetrate through to the internally 
embodied substrates of learning. The work being performed at this data frontier, 
however, is genealogically related to longer-term encounters between education, 
technology, and data science over the last three decades. 


Education Data Science 


A distinctive form of education data science began to emerge in the first decade of 
the 21st century through a series of convergences between technical developments, 
educational agendas, and scientific research and development (R&D) in data science 
and data mining. From the 1990s, concerns with educational accountability and 
evidence-based policy drove the implementation of data systems that could be used 
to record progress towards performance targets and improvement goals (Lingard 
et al., 2013). The use of accountability data as a key source of education policy and 
governance was enabled by large-scale information infrastructures for collecting and 
processing the data (Anagnostopoulos et al., 2013), with interest growing over the 
subsequent decades in analytics packages and data dashboards for analyzing, inter- 
preting, and displaying up-to-date data. Lockyer et al. (2013) state, 


shrinking fiscal resources and the expansion of a global competitive educa- 
tion market have fueled this increasing pressure for educational account- 
ability. The offshoot of these economic drivers has been the development in 
the education sector of standardized scalable, real-time indicators of teaching 
and learning outcomes. 


(p. 1439) 


A key enabling technology in this context was business intelligence systems 
developed for use by commercial enterprises to analyze their organizational per- 
formance through internal data mining, which were adopted by the nascent field 
of Educational Data Mining around 2005 for systematic quantitative analysis in 
the field of education (Agasisti & Bowers, 2017). Specifically, in higher education, 
new forms of institutional or academic analytics were developed to understand 
and improve organizational processes (such as admissions, finances, management, 
and administration procedures; Piety et al., 2014). “Learning analytics” emerged 
at the same time as the analysis of data from learning management systems and 
massive online courses, later expanding to individual-level learner analytics and 
adaptive “personalized learning” software (Siemens, 2013). 

These genealogical points of convergence — between education administration; 
management and business intelligence; accountability agendas; and data science, 
analytics, and data mining — were supported and advanced considerably by a 
growing number of powerful R&D centers located both in academic and 
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commercial settings. Stanford University’s Lytics Lab, the Connected Intelligence 
Centre at the University of Technology Sydney, the Educational Institute of 
Technology at the Open University in the U.K., and the Learning Innovation 
and Networked Knowledge (LINK) Research Lab at the University of Texas at 
Arlington, among others, have become leading global centers of a growing 
international network of learning analytics researchers, developers, and advocates. 
The Society of Learning Analytics Research (SoLAR), a global members associa- 
tion of learning analytics researchers and developers, launched its own journals 
and an annual conference, Learning Analytics and Knowledge (LAK) to define, 
consolidate, and advance the field. 

These labs, centers, and networks have combined disciplinary expertise and 
defined the “skillset” and “mindset” of the figure of the educational data scientist 
(Buckingham Shum et al., 2013). The skillset and mindset of educational data 
science include computer science techniques of computational statistics, data 
mining, machine learning, network analysis, semantic web, knowledge model- 
ling, natural language processing, and human-computer interaction, as well as 
drawing from psychometrics, cognitive neuroscience, and bioinformatics (Cope & 
Kalantzis, 2016; Piety et al., 2014). A 2014 Stanford University report proposed a 
new “specialized” field, combining the sciences of digital data and learning to 
consist of a “big data infrastructure” for analyzing large volumes of educational 
and learning data, plus a “professional infrastructure in the field of learning ana- 
lytics and education data mining, made up of data scientists (straddling statistics 
and computer science) who are also learning scientists and education researchers” 
(Pea, 2014, p. 17). The Stanford report clearly illustrates how an imaginary of 
educational data science was put to work in pushing back new data frontiers in 
educational knowledge production, and in animating the development of both 
infrastructural and professional development to realize that vision. 

These genealogical developments in educational data science, in addition, were 
buttressed and fortified by a growing education technology (EdTech) industry. 
Global edu-businesses such as Pearson were early adopters of learning analytics 
and related forms of educational data science. Pearson (in particular) began a 
“digital transformation” of its entire business around 2012, embedding data ana- 
lytics across its portfolio of e-learning and courseware products, partnering with 
both established and start-up analytics companies, and developing its own in- 
house capacity for data analytics, adaptive “personalized learning” platform 
development, and even the creation of Al-based student assistants (Sellar & 
Hogan, 2019). Wealthy venture philanthropic funders including the Bill and 
Melinda Gates Foundation and the Chan Zuckerberg Initiative became high- 
profile supporters of data-driven education, especially “personalized learning” 
based on the real-time tracking and profiling of students. Zuckerberg even 
seconded a Facebook team to build an adaptive personalized learning platform for 
the Summit Charter School chain in 2014-15, a platform now used by hundreds 
of schools across the United States. 
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The imaginary of educational data science has been further strengthened by 
advocacy from think tanks, consultancies, and international organizations. Think 
tanks such as the Center for American Progress, the Brookings Institution, and 
the Center for Data Innovation; consultancies including KPMG and McKinsey; 
and international organizations including the OECD, World Bank, and the 
World Economic Forum have become highly influential supporters of EdTech 
generally and data-intensive education specifically. These organizations have 
sought to make such forms of education technology policy-relevant to govern- 
ment ministries and departments by linking the capacities of real-time analytics- 
based personalized learning platforms to longer-term agendas of accountability, 
performance monitoring, improvement in educational outcomes, and large-scale 
school and university reform. They have, in other words, sought to legitimize 
data analytics as an authoritative source of governance over education. 

As this brief genealogical survey of the formation of educational data science 
shows, as an emergent field of social, technical, economic, and political relations, 
it has gradually but perpetually pushed back the limits and boundaries of datafi- 
cation in education. Education data science began to challenge those boundaries 
through the application of business intelligence and data mining to organizational 
analysis, but also incorporated increasingly sophisticated analytics based on algo- 
rithmic processing and adaptive machine learning to advocate personalized 
learning as a reformatory vision for whole education systems. Along the way, 
educational data science has assembled both the vast infrastructural capacity and 
enlisted an international network of expert brainpower from across both the 
academic and commercial technology sectors as educational data practitioners and 
professionals. These infrastructures and expert practices are now key to knowl- 
edge discovery in the field of education, with education data scientists increas- 
ingly rewarded with prestigious research grants or commercial positions to address 
complex questions through computational machinery. 


Precision Learning Engineering 


The previous section showed how educational data science has pushed the frontier 
of datafication further into the field of education, from organizational analytics to 
individual-level learner analytics and adaptive learning platforms. Educational data 
science is also advancing an additional data frontier across the very boundaries of 
the human body of the student. As already indicated, educational data science 
draws its expertise not only from computational or statistical disciplines, but from 
psychological, cognitive, neurological, and even biological fields. Within education 
research, the transdisciplinary combination of cognitive science, psychology, and 
neuroscience is known as “learning science,” a field that emerged in the 1990s and 
“marked by collaborations among instructional technologists, educational psychol- 
ogists, content area educators, anthropologists, computer scientists, linguists, philo- 
sophers, and many more” (Evans et al., 2016, p. 1). 
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Originally associated with the established field of AI in education, the learning 
sciences have rapidly gained scientific legitimacy over the past decade (Pea, 2016). 
Learning science approaches based on various aspects of cognitive, neuro-, and 
psy-sciences are being advocated in emerging centers and departments in 
research-intensive universities such as Harvard, Massachusetts Institute of Tech- 
nology, the University of California, Carnegie Mellon, and Stanford, as well as by 
venture philanthropies including the Bill and Melinda Gates Foundation and the 
Chan Zuckerberg Initiative. The OECD has begun to invest significant institu- 
tional resources and advocacy into “the science of learning.” Focusing on “the 
interplay of the biological, physiological, cognitive, and behavioral processes 
supporting the learner,” it advocates for “technological advances, particularly in 
neuroscience, engineering, and computer and information sciences” to be put to 
the task of policy-relevant knowledge production in education (Kuhl et al., 2019, 
p. 16). 

Learning science has also converged genealogically with learning analytics 
research and development, both of which are taken to be key sources of expertise 
in the wider interdisciplinary space of educational data science (Piety et al., 2014). 
One of the key figures in the development of both the learning sciences and 
learning analytics fields clearly marked out this convergence in his Stanford Uni- 
versity report on building the field of learning analytics. Learning analytics spe- 
cialists, Pea (2014) argued, would require skills in “computational and statistical 
tools,” including “traditional statistics skills” as well as “newer techniques like 
machine learning, network analysis, natural language processing, and agent-based 
modelling,” alongside “general educational, cognitive science, and sociocultural 
principles in the sciences of learning” (p. 17). This report also gave an early 
indication of how a learning analytics-mediated version of learning science would 
seek to gather data from students’ bodily movements and facial expressions. Pea 
proposed using data science methods to engage with “non-cognitive factors” in 
learning and the “emotional state” of learners by collecting “proximal indicators 
that relate to learning” through such techniques as “facial expressions detected by 
a computer webcam while learning” (p. 32), plus other data sources like “video, 
eye tracking, and skin temperature and conductivity” (p. 46). 

This turn to capturing autonomic biological signals as “proximal indicators” of 
learning is related to the subfield of emotional learning analytics, and to the 
broader development of “emotional AI” infrastructures and practices in educa- 
tion. Emotional AI, or “empathic media,” is a form of “automated industrial 
psychology” that utilizes biosensing and analytics to monitor signs of emotional 
behavior. McStay (2018) states, 


artificial emotional intelligence is achieved by the capacity to see, read, listen, 
feel, classify and learn about emotional life... this involves reading words and 
images, seeing and sensing facial expressions, gaze direction, gesture and 
voice. It also encompasses machines feeling our heart rate, body temperature, 
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respiration and the electrical properties of our skin, among other bodily 
behaviors. Together, bodies and emotions have become machine-readable. 


p. 3) 


The incorporation of emotional AI into learning sciences and analytics has 
instantiated automated industrial psychology as a mode of knowledge discovery in 
the field of education, rendering students’ bodies, voices, faces, and affects see- 
mingly machine-readable as autonomic proximal signals of the emotional sub- 
strata of learning. 

The combination of learning science and learning analytics has been captured 
in the emerging figure of the “learning engineer.” This term, first coined by 
Carnegie Mellon University’s AI pioneer Herb Simon in the 1960s, has been 
embraced by major technology-intensive universities in the US, engineering 
societies, technology-based philanthropies, and education-focused think tanks and 
consultancies (Lieberman, 2018). The Learning Agency, a startup consultancy 
supported by the Chan Zuckerberg Initiative, the Bill and Melinda Gates Foun- 
dation, and Schmidt Futures, approaches learning engineering as a “revolu- 
tionary” combination of learning science and “the new and powerful tools of 
computer science — rich computation, nascent AI, natural language processing” to 
“promise the same benefits for education research and development” (The 
Learning Agency, 2019, para. 15). The agency adds, 


learning engineers must, then, have a broad knowledge of learning science, 
data science, and computer science. They must be able to understand differ- 
ent aspects of engineering processes, including everything from pedagogy to 
artificial intelligence. The main aim is to use data to improve learning and 
teaching. Learning engineers use technologies, standards, and science to 
propose, test, and implement solutions. 

(para. 22—23) 


Learning engineering is also undergoing a process of standardization and pro- 
fessionalization as a scientific discipline and practice. The Institute of Electrical 
and Electronics Engineers (IEEE), the world’s largest and most influential orga- 
nization for standards-setting in computer technologies and engineering, has 
established the IEEE Industry Consortium on Learning Engineering (ICICLE) as 
a “professional organization committed to the development of Learning Engi- 
neering as a profession and as an academic discipline” (IEEE Industry Consortium 
on Learning Engineering, n.d., para. 2). They define learning engineering as “a 
process and practice that applies the learning sciences using human-centered 
engineering design methodologies and data-informed decision-making to support 
learners and their development” (IEEE Industry Consortium on Learning Engi- 
neering, n.d., para. 1). With membership including more than 60 organizations 
representing industry, academia, and government (including Google, HP, and IBM), 
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ICICLE has established special interest groups to undertake projects in AI and 
adaptive learning technologies, learning analytics, competency frameworks, learning 
data standards, learning data governance, and learning experience design, plus teams 
exploring routes to growth of learning engineering both in industry and academia 
(Wagner & Lis, 2018). 

The Chan Zuckerberg Initiative has become perhaps the most influential 
organization to promote learning engineering, with its director of learning science 
a high-profile advocate of learning engineering as a multidisciplinary blend of the 
learning sciences and learning analytics. Saxberg (2018) states, 


getting the most from learning analytics has to be an interdisciplinary effort: 
computer science, linguistics, education, measurement science, cognitive sci- 
ence, motivational and social psychology, machine learning, cognitive neu- 
roscience among others. These different domains will need to be combined 
to build out an effective evidence-grounded “learning engineering” version 
of learning analytics. 


(p. viii) 


In this imaginary, learning engineers gather and analyze evidence “at multiple levels, 
from clickstreams, motion position data, speech streams, gaze data, biometric and 
brain sensing, to more abstracted feature sets from all this evidence” (Saxberg, 2018, 
p. viii). The use of this evidence across “multiple dimensions” would also allow for 
the identification of “new opportunities for targeted intervention” and “precise 
action” analogous to data-scientific “precision medicine” (p. x). The Chan Zucker- 
berg Initiative, along with Schmidt Future and the Bill and Melinda Gates Founda- 
tion, has begun awarding grants and investments for R&D programs and initiatives 
to support this vision. They include neuroscience-based technologies for tracking 
cognitive processes and brain function, as well as adaptive personalized learning 
platforms based on real-time cognitive and performance assessments. 

The figure of the learning engineer represents an evolution of the image of the 
education data scientist, an expert embedded not just in the infrastructure of 
software, algorithms, and analytics of data science but in the laboratory apparatus 
of the cognitive, neuro-, and psy-sciences too. This configuration of the educa- 
tion data scientist as a learning engineer embodies a particular imaginary in which 
data scientific practices can penetrate human bodies with the precision of neuro- 
technologies such as brain-scanners, body-worn biometric and psychophysiologi- 
cal sensors, and bioinformatics technologies for reading molecular genomic codes. 
This biomedicalized version of the education data scientist possesses not only the 
infrastructural apparatus for making bodies machine-readable, but also the engi- 
neering prowess and design methodologies for programming precision-targeted 
interventions. The precision learning engineer forces the data frontier back 
beyond skin, into the interior psychological, neurological, and biological sub- 
strates of student learning, affect, and cognition. 
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Inscribed Bodies 


The overarching imaginary of education data science is one of automated 
knowledge discovery, whereby computational and analytics infrastructures are 
given key tasks to identify patterns in large and complex datasets under the skilled 
supervision of expert practitioners. Automated knowledge discovery is under- 
girded by a particular rationality and an epistemology rooted in statistical science, 
natural science, and data science styles of thinking and working. This realist or 
empiricist epistemology assumes data provide accurate and objective representa- 
tions of underlying physical or social realities. However, this perspective neglects 
the powerful ways in which infrastructures of data analysis, and the practices of 
those experts embedded in such infrastructures, co-produce and configure the 
data themselves (Borgman, 2019). 

Automated knowledge discovery does not so much discover or reveal facts as 
objective quantitative data but partakes in their construction. 


Choices about methods of producing data... produce and reproduce the very 
objects that they ostensibly reflect. In this sense they are performative in that 
they do not involve the discovery of truths about objects but simultaneously 
represent and enact, that is, bring into being the very objects they are meant 
to describe and represent. 

(Ruppert, 2018, p. 19) 


In the specific context of education data science, Perrotta and Selwyn (2019) 
claim that methods of automated knowledge discovery “superimpose multiple 
layers of algorithmic complexity on stripped-down (and highly contentious) 
understandings of human learning” (p. 4). From this performative epistemology, 
the infrastructures and practices of education data science bring into being new 
understandings of human learning that would not be the same without the 
algorithms, analytics, and automated systems that produced them. 

A consequence of approaching education data science in this performative way is 
to see the data frontier not just as being pushed back into the biological substrates of 
human learning, but to see the frontier as changing the very objects it passes across 
and through. Automated industrial forms of educational data science render bodies 
computable in ways that make them amenable to various forms of intervention or 
even “engineering.” The analogy with precision medicine proposed by the Chan 
Zuckerberg Initiative’s director of learning science is suggestive in this sense. It sug- 
gests a biomedical desire to precisely profile individual bodies through the collection 
and analysis of quantified bioinformation, and then to design personalized, targeted 
“precision education” interventions that might modify, improve, or otherwise 
transform how those bodies function, behave, feel, or think (Williamson, 2020). 

Understood in this way, education data science is not simply a new scientific 
field, but is invested with a particular politics, or a biopolitics of control over 
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human life. Whitehead et al. (2017) conceptualize a novel biopolitics centered on 
the governance of human bodies, minds, brains, emotions, and behaviors. 
“Neuroliberalism,” as they term it, mobilizes insights from the psychological, 
behavioral, neurological, cognitive, and biological sciences “to deliberately shape 
and govern human conduct within free societies” (p. 1). Further, it recognizes the 
role of emotional responses, habits, social norms, and “the automatic, unconscious 
and involuntary within human action, and an ability to be able to predict, 
respond, regulate, enhance and exploit these behavioral vectors” (p. 2). 

With its emphasis on the hidden biological and autonomic signals that indicate 
learning, education data science instantiates neuroliberal strategies of behavior 
change within the field of education. A neuroliberal education data science seeks 
to render the autonomic, unconscious, embodied, and automatic signals that 
indicate learning amenable to being machine-read and analyzed through methods 
of automated knowledge discovery, and from there potentially opened up to 
enhancement and engineering. 

The human subject and body of neuroliberal forms of data science is, in this 
sense, an “inscribed surface of events” understood in genealogical terms, as 
“genealogies are also concerned with the body which is conceptualized as a 
material surface, as a flesh upon which the micro-physics of power leave their 
mark” (Garland, 2014, p. 373). The diverse historical threads that have produced 
education data science as a discipline made up of expert practices and infra- 
structural arrangements in this sense can be understood as imprinted on the bodies 
of its laboratory subjects. Not only are the human subjects of neuroliberal forms 
of education data science inscribed numerically from their data traces and embo- 
died signals — as the result of long-term genealogical convergences between data 
science, education, and learning science — but their analysis also catalyzes practical 
efforts to change the actions, affects, and behaviors of the bodies from which 
those signals and traces were captured. 


Conclusion 


The aim of this chapter was to offer a partial genealogy of education data science. 
By focusing on its imaginaries, infrastructures, and practices, it has illustrated some 
key convergences over the last two decades in the evolving field of education 
data science, from its origins in educational data mining and business intelligence, 
its combination with learning analytics and machine learning-based adaptive 
platforms, through to current convergences with the learning sciences. Education 
data science, like its parent discipline of data science itself, is not a fixed and 
monolithic field, but one that remains in motion as a science in the making. By 
projecting powerful imaginaries of the power of data analytics, education data 
science has expanded and intensified its influence over knowledge production 
across the past decade. It has constructed infrastructures of data collection, calcu- 
lation, and presentation to this end, and developed programs for upskilling and 
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socializing new kinds of multidisciplinary experts who can move across the 
computer, data, and learning sciences. 

Tracing out education data science genealogically reveals the contingency of 
claims to objective knowledge discovery through data analytics infrastructures and 
practices. Instead, education data science constructs the very objects it purports to 
merely measure or reflect. The infrastructures and practices of education data 
science shape processes of learning, as well as learning subjects and bodies, to fit 
into databases and to be made amenable to data analytic inspection. Education 
data science is also imprinted with strong political imperatives regarding perfor- 
mance-based accountability, and in many cases with the market-making concern 
of commercial EdTech businesses for whom data analytics are a source of value 
creation. As with all techno-scientific infrastructures and practices, education data 
science is imbued with politics, economics, values, and disciplinary epistemolo- 
gies. These factors are all part of the genealogy of data science and shape the ways 
in which data about human subjects are brought into being. Much more detailed 
genealogies will be required in the coming years as education data science 
becomes increasingly authoritative and disruptive in educational knowledge 
production. 
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ARTIFICIAL INTELLIGENCE AND 
MACHINE LEARNING 


A Practical and Ethical Guide for Teachers 


Erica Southgate 


Introduction 


In February 2020, I was contacted by a journalist seeking comment on the trial of a 
facial recognition product designed to replace traditional roll call in schools. The 
product had received a significant amount of commercialization funding from the 
Australian Government Department of Industry, Science, Energy, and Resources. 
The published article focused on the lack of Australian regulation related to this type 


ce 


of technology and raised questions about the “convenience of automating roll 
call procedures outweigh(ing) the sensitivity of collecting biometric information” 
(Basford, 2020, para. 7). The article hit several nerves regarding the best use for, and 
ethico-governance implications of, automation powered by artificial intelligence (ATI) 
in schools (in this case, biometric technology that collects information from the stu- 
dent’s body). As the article pointed out, the issue is not only how we can use AI “for 
good,” but for what it is “good for” in educational settings. 

To have informed conversations about the use of AI in education, teachers and 
policymakers must negotiate a dense multi/interdisciplinary web of knowledge 
comprising: technical information about systems and the mathematical and statis- 
tical methodologies that are used by machines to analyze data; ethical frameworks 
that do not necessarily provide definitive answers; and a fast-evolving, intricate 
legal and regulatory landscape. In addition, they must weigh up the potential 
benefits and risks of Al-powered systems developed (typically independent of 
educators) by small and large companies in the national and transnational educa- 
tional technology (EdTech) sector. In response to this complex situation, the aim 
of this chapter is to introduce educators and policymakers to some foundational 
knowledge about AI so there can be critical and productive engagement with the 
technology. This chapter is translational research: it is an attempt to bridge the 
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divide, in an accessible way, between the realms of the technical, the pedagogical, 
and ethico-governance issues associated with AI. I begin the chapter by explain- 
ing AI and its subfield of machine learning (ML) before proceeding to scope out 
current and potential uses of AI in educational settings. I then highlight some key 
ethico-governance challenges for educators and school leaders, policymakers, and 
industry stakeholders. 


What Is Al? 


Every day, in a myriad of ways, we interact with AI. Computing applications, 
such as Internet search engines, smart phone assistants and chatbots, social media 
facial recognition tagging technology and fun filters, ride-sharing apps that 
determine journey pricing, and recommended product advertising, are all pow- 
ered by AI. AI can be embodied in robots (but not all robots have AI) or (often 
invisibly) infused into computing applications and systems. The Organization for 
Economic Cooperation and Development (OECD; 2019) define AI as “a 
machine-based system that can, for a given set of human-defined objectives, 
make predictions, recommendations, or decisions influencing real or virtual 
environments. AI systems are designed to operate with varying levels of auton- 
omy” (para. 12). AI are computer systems that can undertake tasks or activities 
that require features of human intelligence, such as planning, problem solving, 
pattern recognition, prediction, learning from experience, and logical action. AI 
researchers focus on developing machines that can think and behave in a rational 
manner according to mathematical assumptions and rules encapsulated in algo- 
rithms (these are instructions that tell the computer or machine how to accom- 
plish a task or operation), with an increasing emphasis on building capacity for 
automated decision-making. 

The field of AI is decades old, with the term first coined in the 1950s. Over 
time, there has been definitional tension within computer science about what 
constitutes AI. Maini and Sabri (2017) note that standards for what AI is remain 
open to interpretation and continue to change. They suggest as well that AI refers 
to how computers “figure out” and can perform tasks usually done by humans. 
When this occurs, there is a tendency for humans to reject that this is indicative 
of machine intelligence. 


This is known as the AI effect... Perhaps there is a certain je ne sais quoi 
inherent to what people will reliably accept as “artificial intelligence”: So, 
does a calculator count as AI? Maybe by some interpretation. What about a 
self-driving car? Today, yes. In the future, perhaps not. 

(Maini & Sabri, 2017, p. 10) 


Krafft et al. (2020) suggest there are differences between the way technologists 
and non-technologists use the term AI. They argue that AI researchers support 
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definitions that accentuate technical functionality, while policymakers prefer 
more anthropomorphic explanations that relate to aspects of human thinking and 
behavior. Krafft et al. (2020) contend that the problem with this definitional 
mismatch is that functionality-related definitions are more inclusive of current AI 
technologies, whereas definitions that focus on human-like capabilities are often 
more applicable to potential applications. This becomes an issue if the latter 
deflects attention away from existing ethico-governance issues. While public 
education about what AI is, where it is present, and what it can be used for, 
requires an evolving rapprochement between technical and humanist modes of 
communication, it is important to understand why the technology has become 
ubiquitous in everyday applications. 

Since the early 2000s, several developments have enabled the growth in AI 
technology. Advances in computer processing power have facilitated algorithm 
complexity and the ability to distribute and speed up processing across banks of 
computers. This is linked to increased data storage — especially cloud computing — 
with its ability to house and manage large amounts of data (often called “big 
data”) and provide a platform to run ML algorithms. The availability of big data 
harvested from the Internet has facilitated ML at scale, with the growth of open 
source technical communities fostering AI innovation. Specifically, the field has 
benefited from improvements in computer vision, graphics processing, and speech 
recognition technology (Mitchell & Brynjolfsson, 2017). 

Rapid advances in AI and its (often invisible) use in everyday computing have 
given the technology an aura of mystery. As Maini and Sabri (2017) point out: 


artificial intelligence will shape our future more powerfully than any other 
innovation this century. Anyone who does not understand it will soon find 
themselves feeling left behind, waking up in a world full of technology that 
feels more and more like magic. 


p. 3) 


Developing a foundational understanding of AI is vital if we are to reject romanti- 
cized and inaccurate ideas about machine intelligence. There is evidence that both 
children and adults anthropomorphize and overestimate the intelligence of AI in its 
embodied robot and non-embodied (diffuse) forms (see Southgate et al., 2019). 
There are typologies that can assist in dispelling myths about AI. Table 3.1 provides 
an overview of types of AI that currently exist and that may eventuate (although 
there is considerable debate about hypothetical types of Al). 

Table 3.1 illustrates that we are currently in an era of narrow (sometimes 
known as weak) AI. Narrow AI can undertake the specific or focused tasks it was 
designed for. Reactive AI responds to input data but cannot “learn” from past 
experiences to inform decisions. Limited memory AI can accumulate “experi- 
ences” to programmed representations of the world, or “learn” as it goes, and has 
enough “memory” to take appropriate actions. While narrow AI does not exhibit 
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TABLE 3.1 Types of artificial intelligence from Southgate et al. (2019), adapted from 
Hintze (2016) 


Superintelligence AI that exceeds human intelligence in every field. 
Example: None, as this type of AI does not currently exist 
General AI Self-awareness AI at this level would extend the 
Example: None, as this “theory of mind” to predict the 
type of AI does not cur- internal states of others. Having 
rently exist achieved human-like consciousness, it 
might choose to exhibit non-human 
abilities. 
Theory of mind This type of AI would have an upda- 
Example: None, as this table representation of the world that 
type of AI does not cur- includes an understanding that other 
rently exist entities in the world also have their 
own internal states. 
Narrow AI Limited memory AI This type of AI receives current input 


Example: Virtual assistants, 
self-driving cars 


Reactive AI 
Example: AI chess player 


and adds pieces of this input to its 
programmed representation of the 
world. This can change the way the 
AI makes current or new decisions. 


Designed for a specific task, this AI 
receives input and acts on this input. 


They cannot be applied to different 
tasks, and past experiences do not 
affect current decisions. 


the full range of characteristics and behaviors we associate with human intelli- 
gence, it can be equivalent to or outperform humans in specific areas. An exam- 
ple of this was when AI consistently defeated a human champion of the Chinese 
strategy game Go, which had been long considered too strategically complex for 
a machine to master. The next section provides an overview of ML, which is an 
important subfield of AI. Understanding the basics of ML is vital because many 
ethico-governance concerns stem from the way in which it works. 


What Is Machine Learning (ML)? 


A subfield of AI, ML is the science of getting machines to learn and behave like 
humans in an autonomous way by giving them data sets from the real world so 
that their learning can improve over time (Faggella, 2020). The goal of ML is “to 
enable computers to learn on their own. A machine’s learning algorithm enables 
it to identify patterns in observed data, build models that explain the world, and 
predict things without having explicit pre-programmed rules and models” (Maini 
& Sabri, 2017, p. 9). ML is a technically and mathematically complex field. ML 
algorithms are used to classify, generalize, and predict with the accuracy of the 
algorithms improving when trained on greater quantities of data (Burrell, 2016). 
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There are a number of different approaches to ML. For example, supervised 
learning involves labeling or classifying initial input data to train an algo- 
rithmic model to identify patterns. It is supervised because the learning is 
guided by human pre-classification of input data that trains the model to 
make decisions on classification or predictions related to a specific task when 
new data are given to it. Supervised learning is used in applications that sug- 
gest the classification of something, for example, a student might take or 
locate a photo of an animal and the AI application recognizes the animal and 
provides information about it. 

An unsupervised learning (UL) algorithm infers patterns in data that have not 
been labeled (hence being unsupervised) and creates its own structure (sets of 
features) that can be used for pattern detection or classification. This type of ML 
is used to explore and detect patterns when the outcome is not known or pre- 
determined. For example, a company may use a UL approach to detect patterns 
in customer segmentation that were not previously known or understood. It is 
conceivable that with large enough data sets, unsupervised learning algorithms 
could detect patterns in learning behaviors or outcomes that were not previously 
known. 

Reinforcement learning (RL) involves the algorithm interacting in an envir- 
onment to find the best outcome through trial and error without training. The 
AI trains itself based on right or wrong actions to predict best outcomes when 
new data are collected. The best actions are reinforced until the model learns to 
solve the problem or get the best outcome. RL has been used to develop game- 
playing AI and to train chatbots. 

Deep learning (DL) is associated with artificial neural networks (ANNs) that 
are inspired by the way neurons connect in human brains. In DL there are 
numerous layers of algorithms that interact to model data and make inferences. 
DL uses multiple ANNs at lower levels of abstraction to effectively solve chunks 
of a problem and provide these partial solutions to ANNs at higher levels to 
derive a larger solution (LeCun et al., 2015). DL is good for dealing with com- 
plex data, such as natural language processing, which involves complicated voca- 
bularies or visual processing that has intricate pixel information (Maini & Sabri, 
2018). DL could be used in a smart classroom that had sensors that input audio- 
visual data over time. DL would have different ANNs working on recognizing 
aspects of the biometric information from teachers and students (e.g., torsos, 
limbs, eyes, noses, mouths, fingers, palms, and voices). Other ANNs would use 
the results from these ANNs to classify the types of verbal and non-verbal 
exchanges, and yet other ANNs could process these results to detect patterns in 
non-verbal student interaction and between the teacher and students. This could 
then be fed into a dashboard to give teachers and learners insights into classroom 
interactions. While this is an imaginary example (and one that has ethical impli- 
cations to be discussed later in this chapter), it is illustrative of how DL could 
process large amounts of complex data from the classroom. 
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It is important to grasp the basics of ML for several reasons. Some of the rea- 
sons include the need for humans to be able to identify when they may be 
interacting with an Al-powered system so it does not seem like “magic,” but 
rather the result of complex mathematical models that automate responses often 
designed to “nudge” or influence people towards particular actions. As ML 
usually needs big data, sometimes invisibly harvested in real time as people 
interact with machines, questions need to be asked about privacy and consent. 

Insights derived from ML are based on the data that algorithms learn from, and 
these can have biases or provide superfluous insights, even when harvested in real 
time. Who hasn’t received a product recommendation on their social media feed 
that was totally inappropriate? Sometimes, the humans who choose data for ML do 
not check for or recognize the biases in the data because they are from a dominant 
social group; this ML generated bias can impact people’s lives. For example, research 
has shown that facial recognition technology can produce a higher rate of incorrect 
matching (false positives) for darker skinned people than for lighter skinned people. 
This can cause problems with biometric passport systems and law enforcement 
authorities who are increasingly depending on the accuracy of the technology to 
identify suspects (Drozdowski et al., 2020). These are only a few instances of AI bias. 

ML can also be opaque or a “black box” in the sense that the algorithms that 
make decisions are proprietary (the property of companies who will not open 
these for inspection) or so complex in their operation, like ANNs, that even the 
scientists who develop the systems are unable to completely understand the pro- 
cesses through which the machine makes its decisions (Martin, 2019). When ML 
affects life opportunities — i.e., when it is used to determine if someone gets a job 
interview, a debtors notice, a loan, arrested, or has access to a particular curricu- 
lum pathway in an computer tutoring system — serious ethical and governance 
issues arise. These issues will be discussed later in this chapter. 


How Is Al Used in Education? 


It is useful to think about how AI is used in education from several angles. At the 
level of schooling systems there is the integration of Al-powered products that are 
used to streamline or automate administrative procedures. There is also an interest in 
e-learning platforms and learning management systems that generate data that can 
use ML to provide organizational and classroom insights, for example, to describe 
and predict trends in academic success and provide early warning signals for students 
who may not be achieving expected academic outcomes (Kavitha & Lohani, 2019; 
Matetic, 2019; Nam & Samson, 2019). Higher education has led the way in using AI 
to understand student engagement in online learning and student attrition. Admin- 
istrators of school systems are no doubt looking to the utility of AI with (sometimes 
contentious) moves to automate standardized testing (ABC News, 2018). 
Understanding the technology’s relationship to education means recognizing 
the growing ubiquity of Al-powered products not generally marketed under the 
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EdTech banner. Leaving aside the most obvious application (search engines), a 
range of other products used in schools are infused with AI. These products 
include: Adobe Photoshop, video products that use ML to facilitate the easy 
selection and altering of objects, collaboration applications such as Microsoft 
Teams that have chatbot functionality, and teleconferencing platforms that use AI 
to suppress unwanted noise during calls. 

Beyond these products designed for more general markets, there is a specialist 
field called AI in education (AIEd) which has been around since the 1970s (see 
Isotani et al., 2019). AIEd’s goal is to enable more personalized, flexible, and 
engaging learning, and to automate mundane teaching tasks such as assessment 
(Gulson et al., 2018; Luckin et al., 2016). There are large multinational corpora- 
tions (such as Pearson) and smaller EdTech companies driving AIEd development 
and research at universities — some in partnership with industry — to investigate 
the efficacy of AI -powered applications for learning. AI is also used in a business 
intelligence sense for the production of learning analytics (the type teachers might 
see on a class dashboard) and in mining the data collected by educational orga- 
nizations to provide insights into how people interact with systems. 

Although wide-spread integration of AIEd is currently limited, there are several 
applications that have been continually under development. Intelligent tutoring sys- 
tems (ITSs) simulate human tutoring by providing timely feedback, personalized 
instruction, and curriculum pathways for students without human intervention or 
with only targeted teacher oversight (Luckin et al., 2016). Some research suggests 
that ITSs have positive effects on learning, similar to that produced from real teacher- 
student interaction (du Boulay, 2016; van Lehn, 2011), with equivocal evidence of 
learning efficacy for college students (Steenbergen-Hu & Cooper, 2014). Further, 
mathematics ITSs for K-12 education can be beneficial for students with existing 
self-regulatory skills (Steenbergen-Hu & Cooper, 2013). 

AIEd has sought to develop pedagogical agents (PAs), which are digital or 
virtual characters integrated into learning technologies. PAs display as human and 
animal avatars (or other types of virtual characters) and have instructional, moti- 
vational, and communication purposes (Johnson & Lester, 2016; Kim & Baylor, 
2016). Interaction with PAs can be written or spoken, or in virtual worlds, 
through non-verbal communication. They can deliver information, demonstrate 
and model learning, coach, guide, and scaffold the student, and facilitate assess- 
ment. Students can even teach PAs. At present, PAs are mainly used for infor- 
mation delivery or for giving directions (such as chatbots). Evidence regarding the 
effectiveness of PAs is mixed, with one meta-analysis finding that they are prob- 
ably more effective in K-12 than in tertiary education (Schroeder et al., 2013). 

The Internet of Things (IOT) is a term used to describe the increasing capacity 
for common objects to connect to the Internet and interact with other devices 
(Timms, 2016). IOT has prompted interest in what “smart” classrooms of the 
future might look like. While mainly theoretical at present, smart classrooms 
could include interconnected devices and sensors that interact to automate 
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functions such as security systems, climate control, and energy consumption. In 
addition, personal and professional computing devices, learning platforms, and 
administrative systems can connect to harvest and analyze data for learning 
insights (Li et al., 2015). This vision of schooling usually includes the collection of 
biometric data, such as the facial recognition technology example mentioned at 
the beginning of this chapter. 

Biometrics is the automated collection of information related to the body that 
is analyzed using ML. Biometric data can be biological, including facial and voice 
recognition, fingerprints, iris patterns, heart rate, body temperature, or perspira- 
tion. It can also be behavioral, such as vocal patterns, eye tracking/gaze attention, 
gait tracking, or typing recognition. There is also the experimental use of the 
brain-computer interface that uses electroencephalography (EEG) devices to 
measure brain activity. Biometrics uses these unique physical characteristics for 
verification or identification purposes (Royakkers et al., 2018). Biometric data are 
not just information about a person — i.e., the type of information you share 
when setting up online accounts — but information directly of the person. 

Affective computing is a controversial yet burgeoning area that uses a range of 
data types, including biometrics. Machine vision and natural language processing 
technology is used with ML to “identify” emotions so that applications can 
respond to a user in real time. There has been critique of this on several grounds, 
including: the inability of ML mathematical models to grasp the cultural and 
contextual specificity of emotional facial expressions (Barrett et al., 2019); the 
racially biased labeling of data sets used in supervised learning to train AI (Rhue, 
2018); the lack of representation of children’s data that leads to poor prediction 
outcomes (Bryant & Howard, 2019); and the ethically dubious use of technology 
to manipulate or “nudge” people’s behavior without their knowledge or consent. 
Moreover, there are persuasive arguments that the use of biometric technology in 
EdTech is currently without evidence of efficacy for learning, and that it con- 
travenes children’s human rights (McStay, 2019). 

Since most AIEd applications are pioneered in higher education, it is worth con- 
sidering the findings of a recent systematic review. Zawacki-Richter et al. (2019) 
reviewed four areas of AIEd: profiling and prediction, assessment and evaluation, 
adaptive systems and personalization, and intelligent tutoring systems. They found a 
general lack of critical reflection of the challenges and risks of AIEd, a weak con- 
nection to theoretical pedagogical perspectives, and a need for more extensive 
research on the ethical implications of AIEd. It is to this last topic that we now turn. 


What Are the Ethical Concerns with Al and Some of the 
Governance Implications? 


The last decade has seen much international activity on developing an ethics of 
AI that can underpin its design implementation and governance (Cath et al., 
2018). Dignum (2018) captures major ethico-governance concerns by asking: 
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What does it mean for an AI system to make a decision? What are the moral, 
societal and legal consequences of their actions and decisions? Can an AI 
system be held accountable for its actions?... How can these systems be 
controlled once their learning capabilities bring them into states that are 
possibly only remotely linked to their initial, designed, setup? Should such 
autonomous innovation in commercial systems even be allowed, and how 
should use and development be regulated? 


(p. 1) 


Education has been designated as a sensitive domain where automated mistakes 
can significantly damage humans (Campolo et al., 2017). I have previously 
explored the ethics of AI in education and its human rights significance in depth 
(see Southgate et al., 2019). The following is an abridged version of that work, 
with a focus on five key pillars that should underpin the design, use, and gov- 
ernance of AI in educational settings. 


1. Awareness 


There is a need for students, teachers, administrators, and policymakers to develop 
foundational knowledge of: what AI is, what it can and cannot do, where it is 
present (especially if it is invisibly infused in systems), how it operates (including 
algorithmic “nudging”), how it is collecting data (and for what purpose and what 
privacy and consent issues this entails), and its impacts on humans and the broader 
environment. School curriculum needs to raise awareness of these issues. Teacher 
professional learning based on independent expert advice on evidence of ethical 
use cases, legislation, regulation, and guidelines is required. Awareness is one foil 
against deceptive and malicious use of the technology. 


2. Explainability 


The very basis of education is the ability of educators to explain content knowl- 
edge and pedagogical and pastoral actions and their consequences. A vital peda- 
gogical project lies ahead of the teaching profession. It involves the sustained 
development of formal public and school-based explanations about what AI is 
and its implications for education and broader society, including the world of 
work. As educators, we must be able to explain why an Al-powered system made 
its decision and how it affected student learning positively or negatively. For 
example, if ITSs are to be widely deployed, teachers must be able to explain why 
a student was directed along a particular curriculum pathway by the machine, 
what implications this had for student learning as opposed to other options, and 
the broader impacts on diverse groups of learners. It is vital that school commu- 
nities ask ethical and pedagogical questions about AI products before they are 
used in schools, so industry can answer these questions in an accessible and honest 
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manner. Schools (and even school systems) can lack access to independent tech- 
nical advice required to ask relevant questions and enable comprehension of 
answers about AI systems. There is a profound difference in knowledge about Al 
between the schooling sector and the computing industry. Addressing this is vital 
for good governance. 


3. Fairness 


AI has the potential to radically disrupt the world of work and to exacerbate social 
inequality. This raises questions about the role of schooling in addressing 
inequality in an AI world, especially when digital inclusion is a major issue in 
many schooling systems. The benefits of AI for learning need to be evenly dis- 
tributed. The digital rights of the child in this age of “datafication” and “data- 
veillance” need serious attention at curriculum and policy levels (Lupton & 
Williamson, 2017). To date, much of the bias, discrimination, and harm pro- 
duced by algorithmic and automated systems have not been discernible until a 
significant time after their implementation, when adverse patterns have emerged. 
Significantly, it may not be possible to detect when harm is occurring, because it 
cannot be assumed AI will discriminate in similar ways as humans (Wachter et al., 
2020). Schools need to understand and respond to this with new and tried 
ethico-governance measures. 


4. Transparency 


This area has received significant attention in computer science and philosophical 
literature. Winfield and Jirotka (2018) state that “an important underlying prin- 
ciple is that it should always be possible to find out why an autonomous system 
made a particular decision (most especially if that decision has caused harm)” (p. 
8). The opaque nature of some AI means that it can be invisibly infused into 
computing systems in ways that influence our interactions, decisions, moods, and 
sense of self without us knowing (Cowie, 2015). The proprietary nature of Al 
products creates a situation in which industry is reluctant to open up the algo- 
rithmic workings of the product for public or independent scrutiny (Burrell, 
2016). This creates a situation in which customers must rely on industry assur- 
ances rather than independent assessment. The opaque nature of some types of 
ML means that developers can use “black box” components in their software, the 
functioning of which they often do not fully understand. Deep learning, which is 
driving advancements in AI, has created a situation where developers will likely 
build AI systems that cannot always be guaranteed to operate exactly as intended 
in every circumstance. In addition, they may produce results that are unable to be 
fully inspected, validated, or justified by ordinary means, which leads to increased 
risk of undetected or unforeseen errors, biases, and harms (Institute of Electrical 
and Electronics Engineers [IEEE], 2018). 
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5. Accountability 


Governance of and accountability for AI in education entails a multilevel approach 
that includes community consultation, development of mechanisms for ongoing 
contestability of the use of such systems, and policy development and risk assessment 
related to national and international laws, regulation, and standards. To be clear, the 
speed of AI innovation has largely outpaced governance structures and traditional 
policy levers with a significant technical and information differential existing 
between a small coterie of wealthy and influential multinational players (who mainly 
develop the technology and bodies that devise governance mechanism), and schools/ 
school systems that procure products. This has called into question the fundamental 
ability of governments to regulate the technology in traditional ways (Guihot et al., 
2017). There is the real possibility of regulatory capture in this space, which occurs 
when those in governance positions, whether in schools or bureaucracies, become 
dependent on potentially conflicting commercial interests for advice. The depen- 
dence on industry to provide most teacher professional learning in the EdTech space 
attests to how rapidly the influence of industry has grown. There is often talk about 
“trade-offs” with AI — of convenience, cost saving, and/or efficiency — however, 
how can trade-offs be considered with probity by teachers, school leaders, and pol- 
icymakers without access to independent experts with commensurate depth of 
knowledge that exists in industry? 

The IEEE (2018) elucidates on the difficulty of accountability in an AI world. 
They highlight how the complexity of autonomous and intelligent technology 
presents inherent difficulties for system users to understand the potential and 
limitations of such systems, even as they interact with them (p. 27). This opacity 
and the decentralized character of development and function tend to “complicate 
efforts to determine and allocate responsibility when something goes wrong with 
an AI system. Thus, lack of transparency both increases the risk and magnitude of 
harm (users not understanding the systems they are using) and also increases the 
difficulty of ensuring accountability” (p. 27). 


Conclusion 


Al-powered systems pose different levels and types of risks; some minor, others 
more significant, and some hazards and harms may not be identified until long 
after implementation. It is incumbent on educators to grapple with this uncer- 
tainty and equip themselves with knowledge so that they can decide when AI- 
powered systems might be beneficial, where they might be worthy of being 
trialed, and where they should never be used. This means substantially democra- 
tizing conversations about AI so that the teaching profession and whole school 
communities can be involved in its design, implementation, and governance. 
There is exciting work going on in the area of AI literacy (Long & Magerko, 
2020); however, this should always be under the broader domain of digital 
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literacy and be cross-curricular in its approach for both students and educators. All 
students, their teachers, and preservice teachers should be learning about AI so 
that the technology is not viewed as magic. Schools and schooling systems need 
access to independent expert advice not only to deepen technical understanding 
but to guide them through evidence of efficacy for different types of learning and 
the ethico-governance implications of the technology. Consideration of AI- 
powered systems includes, but goes well beyond, usual considerations of consent 
and privacy. We require whole-of-school-community dialogue about the role of 
automation in education, what AI is and what it is good for, and when it may be 
too risky or dehumanizing to use. While this dialogue must include industry and 
government, it should be led by teachers and students. They should ultimately be 
the architects of AI in education. 
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THE RELATIONSHIP BETWEEN 
HUMANS AND MACHINES IN 
PUBLIC POLICY 


Brian Lee-Archer 


Introduction 


The digital economy is giving rise to a challenging paradox. Societies are lever- 
aging digital data to help address the most wicked problems facing the planet, 
such as climate change, poverty reduction, and public health outcomes. In parallel 
to this public value creation, there is a growing gap in trust between citizens and 
their governments, and the use of digital data is a significant contributing factor. 
In an environment of falling trust in public institutions (Organization for Eco- 
nomic Cooperation and Development [OECD], 2018), support for public policy 
initiatives leveraging digital technology to deliver better educational outcomes for 
children is dependent on the human element in decision-making. As digital data 
moves through the data lifecycle of research, policy, program delivery, and eva- 
luation, the way humans work with machines is vital to building public trust. 
This chapter examines the emerging relationship between humans and machines 
(often referred to as human+machines) in a decade dominated by artificial intel- 
ligence (AI) and the use of big data (Daugherty & Wilson, 2018). It is proposed 
that it will be innovation from within the human dimension, guided by a 
dynamic base of digital data, which provides the balance between managing risk 
(e.g., privacy, security, and ethics) and rewards (such as better educational out- 
comes for stronger societies). Stronger societies use existing assets, including their 
data, more effectively and promote citizen-led initiatives to enable them to take 
control of their affairs in a democratic and inclusive way (Russell, 2009). 

Trust in public institutions and government is constantly challenged. Rises and 
falls in public trust come as no surprise, given that political leaders, governing 
parties, and public administrators come and go. Rather than getting caught in 
fluctuating public trust, policymakers can progress digital adoption programs by 
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paying close attention to building public confidence in each digital initiative. A 
structured, process driven approach to AI and big data provides incremental 
building blocks for growing public confidence. Making the public aware of the 
way in which humans work with machines in an augmented fashion, will also 
contribute to building confidence in digital adoption. 


Digital: It Is Not New, and It Is All around Us 


Digital technology has become synonymous with breakthroughs in addressing 
societies’ most wicked problems, such as poverty alleviation and rising inequality, 
the effects of climate change, chronic and life-threatening health conditions, and 
providing stability within global financial markets. Fundamental to addressing 
these problems is the on-going investment by society in human capital through 
education. As every aspect of human activity is increasingly influenced by digital 
adoption, education systems are called upon to supply skilled human capital. In 
keeping up with the demand for the digital skills of the future, the sector is 
challenged to digitally adapt at a similar rate of change. 

Digital technology today is largely characterized by the rise of AI. Al is a collection 
of technologies that combine data, algorithms, and computing power (European 
Commission, 2020), and it is moving rapidly beyond the realms of science fiction 
into mainstream, everyday life. Phenomenal advances in computing power over the 
past two decades to put information in the palm of our hands (through smartphones), 
and the vast amounts of digital data generated every day, serve as catalysts for Al’s 
potential to change the way we live, work, learn, and play. 

While this rapid change occurs around us, we should reflect on the fact that, in 
2019, the world celebrated the 50th anniversary of the Apollo 11 moon landing. 
While it was an incredible feat of human engineering, courage, persistence, and 
ingenuity, the role of digital technology in making the mission successful is less 
well known (IBM, 2011). At times, it may seem that digital technology is new 
and unique, particularly because the information technology industry is adept at 
making new products, software, or hardware appear far superior to whatever 
came prior. However, technology has been evolving for decades. 

Digital technology has contributed to transformations across every industry 
sector, leading to rising living standards around the world. From the Apollo 11 
moon landing through to the launch of the first smartphone — the Apple iPhone 
(Apple, 2007) — in 2007, it has been an upward trajectory of widespread value 
creation through digital technology. If there were any risks or downsides to the 
technology as it evolved, they were largely hidden or set aside by policymakers, 
and overshadowed by public enthusiasm for new technology. 

The recent rise of AI — in particular, the way in which it exploits a person’s 
digital footprint — is shifting public opinion. Consequently, the positive image of 
digital technology is now under threat as the risks become more evident and 
potential misuse of personal data by governments and commercial entities 
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becomes a significant public policy issue. As a result, the discussions about risks in 
the ways personal data are used are likely contributors to falling levels of trust in 
government. One such example of data privacy concerns was evident in the 
COVID-19 pandemic, when several countries sought to contact trace the virus 
through an application (app) on smartphones (Meixner, 2020). As a public health 
and economic recovery initiative, it attracted significant public concern around 
data privacy. Falling levels of trust in government were likely contributors to 
people’s concerns in how their personal data might be misused. These concerns 
act as an inhibitor to the take-up rate of the app in a potentially disproportionate 
manner to the potential societal benefits. 

AI and big data provide opportunities for health, social, economic, and envir- 
onmental domains to be addressed in ways never previously imagined. As society 
meets these challenges with insight gained from the linkage and sharing of digital 
data, the data related risks must be considered in parallel. Further, the ethics of 
responsible use of AI and the moral hazards of automated decision-making must 
also be addressed. It is no longer appropriate to stand back and marvel at the 
information technology industry. Without extensive public debate on how these 
technologies affect each member of society, people could end up living in world 
they do not want and/or were not expecting. 

The COVID-19 pandemic has provided society with a sneak peek into the 
future as digital technology became the bedrock of human communication for 
business, education, and simply keeping in touch with family, friends, and col- 
leagues. Not all experiences using such technology are currently productive or 
value adding, and there are downsides to this forced dependence on digital 
technology. While most people would agree that no technology is not an option, 
too much, especially without human intervention, is not good either. There is 
now an opportunity, after this glimpse into the future, to aim public policy 
towards a happy medium where people and technology work together in a 


complementary manner. 


BOX 4.1 SELF-DRIVING OR AUTONOMOUS VEHICLES - THE 
HUMAN DIMENSION 


In recent years, there has been extensive research and development into self- 
driving or autonomous vehicles. While significant progress has been made, 
moral and ethical dilemmas associated with Al and automated decision- 
making stand-out. 

Consider the following proposition: The Al within an autonomous vehicle 
on a public roadway detects a potential collision with a pedestrian pushing a 
pram. There are two options available to avoid a fatal collision: (1) swerve into 
oncoming traffic, or (2) drive onto a footpath where an elderly couple are 
standing. Both options will potentially kill or seriously injure one or more 
people. The machine must decide what to do. Which option has more or less 
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risk of causing death? Is one life more valuable than another? Should the 
machine prioritize a child’s life over an elderly person’s? Does the machine 
give priority to the people in the car it is controlling? 

It may be possible to layout a rational decision-making process for Al to 
follow. The machine learning algorithm could learn and modify itself based 
on learnings from the experience. Would this be considered a responsible use 
of artificial intelligence? More importantly, is this the right question to ask? Is 
it a responsible use of artificial intelligence to allow an autonomous vehicle to 
travel on standard public roadways, shared with pedestrians and vehicles that 
do not have complementary technology? The most likely answer is no. With- 
out significant changes in how roadways are constructed with pedestrians 
and cars kept apart, this hypothetical scenario for the responsible use of Al 
should be kept hypothetical. 

It is the human element that makes the decision on where and how 
autonomous vehicles can be safely deployed. Mass transportation is an 
example where this is occurring. The Sydney Metro line from Tallawong to 
Chatswood, New South Wales, is a fully automated and driverless rapid transit 
system that opened to the public on 26 May 2019 (Noyes, 2019). It is human 
decision-making that deployed this automated metro system on a standalone 
rail system. 


The term “digital technology” can be thought of as automation where 
machines perform the tasks humans are not good at (such as repetitive tasks) and/ 
or are performed repeatedly. At the other end of the human+machines scale are 
tasks such as counseling and personal care that are best performed by a human. 
There are many activities in the middle of these two extremes where machines 
can augment human capability (Daugherty & Wilson, 2018). This middle ground 
is the domain of AI. 

Al is rapidly emerging as the transformative technology of the information age. 
There are many risks associated with AI, and these are often expressed in terms of 
fears, hopes, predictions, and sometimes dire warnings (for example, “the 
machines are taking over”). AI, however, is a 60+ year-old technology. Since the 
term “artificial intelligence” was coined in 1956 (Tate, 2014), the fantasies of 
science fiction and a world controlled by robots have often found their way into 
the public discourse. AI has come to the fore in recent years as phenomenal 
advances in computing power, combined with the availability of data, have given 
rise to machine learning. 

Machine learning is an application of AI where systems can automatically learn 
and improve from experience without being explicitly programmed (Hao, 2018). 
Machine learning focuses on the development of computer programs, in parti- 
cular algorithms, that access and use data to learn for themselves, thereby 
improving algorithms without direct human intervention. Machine learning 
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offers new ways to solve problems by developing and/or modifying existing 
algorithms and then training the machine with data captured and observed from 
the real world. 

Society has reached a phase of rapid expansion in capability in addressing big 
problems through the use of AI and machine learning. The potential benefits 
within public administration are expanding at an incredible rate; however, so too 
are the risks, including the misuse of personal data, ethics, cyber security, algo- 
rithmic bias, and the moral hazards of decision-making with AI (Lee-Archer 
et al., 2016). This new class of risk is troubling for public administrators and the 
public alike. It is the “without human intervention” aspect of machine learning 
and AI that gives rise to some of the most troubling predictions of the unintended 
consequences of AI. 

In recent years, the terms “responsible AI” and “responsible use of AI” have 
risen to prominence (Dignum, 2019). The ever-increasing volume of personal 
data coming into the hands of government is a source of great power, and with 
that comes great responsibility in ensuring the data are managed correctly and are 
used to create public value. From addressing national security issues to delivering 
better health and educational outcomes, public value derived through leveraging 
personal data at an individual or group level is contingent upon the level of 
confidence people have in the institutions charged with managing the risks (see 
the information on “Self-driving or autonomous vehicles — the human dimen- 
sion” for an example of responsible AI and ethical considerations). 

While there will always be demands for fail-safe guarantees regarding data 
security, these are, in the main, unrealistic. In the same way that the airline 
industry cannot guarantee an accident will never occur, the industry has a track 
record that gives the traveling public confidence in the safety record of individual 
airline operators. Establishing high degrees of public confidence for the airline 
industry to function effectively is dependent upon a structured approach to 
managing risks and demonstrating how the residual risks are outweighed by the 
potential benefits. 

In terms of public policy, the Australian National Audit Office (ANAO), when 
examining governance mechanisms in the Australian Public Service to manage 
risk, stated the importance of establishing positive risk cultures in agencies where 
risk appetites and tolerances are clearly defined and communicated to staff. This 
sets up a paradoxical situation, as while the public generally expects some mana- 
ged risk-taking by administrators as they improve services through innovation, 
there is a low tolerance for failure or error. The connection with innovation was 
also identified, when Grant Hehir (2018) — Auditor-General for Australia — 
stated, “positive risk culture is also an enabler of innovation. If staff are afraid to 
fail, they are unlikely to take calculated risks and be innovative” (para. 4). 
Encouraging staff to think outside the box but not be afraid of an acceptable level 
of risk when it comes to innovation, with the support of sound risk management 
processes, helps create enduring innovation for better policy and service delivery. 
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Rather than a negative hurdle that cannot be overcome, the presence of risk 
(when managed well) is essential to progress through to the stage of building 
confidence. Responsible use of AI in addressing the risks does not mean aban- 
doning human effort or intervention. Instead, the way to manage the risks is to 
use machines to add, enhance, and complement the ability of people. While AI 
attempts to improve human effectiveness, an effective AI system cannot exist 
without human effort. It is this human and machine dimension of decision- 
making — where the potential benefits of AI are realized, and where the risks are 
known and managed — that is most powerful. 

Despite concerns surrounding the use of AI, there are examples where humans 
have high levels of confidence in machines operating autonomously. One such 
example is the marking of multiple-choice exam papers using scanning technol- 
ogy and a set of rules to judge the accuracy of each answer. Conversely, there are 
areas where some people believe humans should be in complete control, such as 
evaluating a creative writing piece, because of the number of elements involved 
in discretionary decision-making; for example, an examiner’s assessment of the 
language elements of a creative writing piece against a rubric. 

These are things a machine could learn if trained by a human. With a piece of 
creative writing, AI could read the text and provide an assessment of the language 
conventions or elements used. A responsible use of AI would leave the final 
decision on the creativity of the piece within the domain of the human exercising 
judgement, but they would be guided by the automated assessment. In the mul- 
tiple-choice assessment, a human could also guide machine learning algorithms to 
recognize the various ways students may annotate or color a form. 

The important principle at play in these examples is that AI augments, rather 
than replaces, human decision-making. Daugherty and Wilson (2018) explained 
in their book “Human + Machine”: “humans and machines aren’t adversaries, 
fighting for each other’s jobs. Instead, they are symbiotic partners, each pushing 
the other to higher levels of performance” (p. 106). Human oversight and gui- 
dance on where machines are used and how they are trained provide a basis for 
improving human decision-making within public policy development and 
administration. 


Problem Solving through Al 


AI is a powerful tool to augment human decision-making in analyzing and 
understanding the complex interactions between social determinants, which are 
addressed through the various arms of public policy. Through AI and machine 
learning, patterns (or correlations) can be identified providing the clues and 
potential pathways to addressing causality within intersecting public policy 
domains. Public administrative systems, such as education, are comprised of 
countless decisions across many actors, including policymakers, bureaucracies, 
school administrations, teachers, parents, guardians, and students. From policy to 


Humans and Machines in Public Policy 81 


program design, regulation and education services and delivery, actors are inno- 
vating with digital technology to create value. As organizations use digital tech- 
nology to solve complex business problems, they have to plan for the inevitable 
social and economic impacts of this value creation (Fjord, 2020). 

Creating public value is the intended outcome from investing in digital tech- 
nology by the administrative arms of government (OECD, 2014). Creating value 
comes via “digitization” (doing the same or similar things via digital means) and 
“digitalization” (using digital means to do things differently; Chair in Digital 
Economy, 2017). As in previous industrial revolutions, the digital economy 
involves the destruction of existing systems and structures, while driving a pro- 
found transformation in the world of work and the labor market as a whole 
(Palier, 2019). 

A digitalization mindset is where public policymakers place the creation of 
public value ahead of technology adoption and deployment. For example, AI can 
be deployed in the public interest to reduce the costs of education services by 
augmenting human decision-making in allocating highly skilled and expensive 
resources (European Commission, 2020). A digitization approach, on the other 
hand, seeks to use technology to automate for efficiency, without solving a real 
business issue. This approach has the potential to achieve minimal benefits at a 
potentially high cost and is sometimes referred to as “lipstick on a pig” (Waller & 
Weerakkody, 2016, p. 19). This represents a digitization drive for efficiency 
without necessarily enabling an innovation led transformation, as policy instru- 
ments remain largely unchanged. Within the education sector, an example of this 
approach is making existing course work available online. Thinking creatively 
about how technology is used to transform education delivery leading to better 
outcomes (such as addressing distance learning challenges rather than making 
learning available online) is where true innovation occurs. 

Addressing complex problems, such as educational disadvantage across socio- 
economic cohorts, requires the capabilities of new technology to draw insights 
from the already large and growing quantities of data, in order to influence 
decision-making. Rather than replacing humans, the aim of digital technology 
and AI is to provide people with “superpowers” when making decisions, by 
providing insight into what does and does not work (Daugherty & Wilson, 
2018). To succeed, business leaders need to commit to designing for human 
intelligence and optimizing the relationship between people and machines. 

The traditional approach in implementing public policy through administrative 
systems is a one-size-fits-all approach. It can appear more straightforward in 
defending the perceived fairness of a standard service offering. However, this can 
lead to over servicing and under servicing. 

Digital technology can enable better outcomes that are aligned with policy 
intent through mass personalization. The calibration of the service response can 
be tailored (including in real time) according to the context of the circumstances 
and the capacity of the actor in delivering the services. In education, this does not 
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mean creating individual learning solutions for every person; AI assisted decision- 
making can determine where tailoring standard offerings can be most effective 
and efficient. To solve problems by augmenting human decision-making cap- 
ability, AI and machine learning systems need to ingest vast amounts of digital 
data. Much of the data relate to people and their circumstances. Digital data, by 
its very nature, can be easily and rapidly misused (at scale, affecting whole popu- 
lations or cohorts) with potential for profoundly negative impacts. Solving com- 
plex problems with digital data can be a risky business. 


Building Confidence to Address the Risks 


As people’s expectations rise for digital service quality and experience through 
the responsible use of their personal data, there is a growing gap in trust in 
many countries between citizens and their governments. In fact, within OECD 
countries, only 43% of people trust their government (OECD, 2018). There are 
many factors contributing to this rising distrust, not least of which is an 
increasing power imbalance in favor of government, as public administrators 
enhance the capability of using digital data to inform administrative decision- 
making. When people are surveyed on their trust in government, their answers 
might reflect a one-off experience or be an amalgam of many experiences. 
Their views may reflect their most recent experience with the local public 
school, a collection of experiences with the health system over an extended 
period, their opinions on the current state of the political process, or any com- 
bination thereof. 

Given the varied nature of the experiences that impact one’s trust in govern- 
ments, it can be difficult to ascertain the levels of trust in the broader govern- 
ment, or with a specific arm of government (such as an education system). Trust 
is often considered within a dimension of time, as it matures within an enduring 
relationship. People are likely to experience periods of satisfaction and displeasure 
with their government or an arm of the bureaucracy. Throughout this period, 
there may be a sense of overall trust, or they may inherently never trust the 
institution. Some people, perhaps a sizable cohort, will hold largely ambivalent 
attitudes to the institutions of government, including the school system. These 
people can be largely influenced by the circumstances of the moment. 

When considering adoption and acceptance of digital services, such as AI 
assisted decision-making, a better place to start is to focus on building confidence. 
Confidence can be more easily expressed in the moment, and as contextual and 
value-based, for example, “how confident do I feel about this transaction or 
activity being successful and will it do me any harm?” A low level of confidence 
may be sufficient for a digital transaction or data sharing if the perceived value 
outweighs the risk. 

Every experience a person has with a public entity can be considered a 
moment of truth in terms of confidence in the competency of the entity. Jan 
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Carlzon, CEO of Scandinavian Airlines, turned the airline industry service 
experience around in the early 1980s by focusing on every interaction with a 
customer as a moment of truth. Public entities can adopt a similar approach to 
building people’s confidence in how their data are used at each moment of 
(digital) truth (Carlzon, 1987). A positive moment of truth in the digital realm 
might be the ease and convenience of registering for a new learning experience. 
It might be a personalized website experience or a simple transfer to a human 
operator for advice. Conversely, a poorly designed digital experience that raises 
concerns that personal data has been compromised and/or is being used for a 
purpose other than what consent has been provided would be a negative moment 
of truth. 

The digital realm is providing new means for public administrators to create 
moments of truth by meeting people where they are at in terms of their need, 
capability, and readiness to engage in the digital environment. Rather than 
aiming for high expectations of an enduring relationship of trust, public agencies 
can be better served by focusing on the fundamentals of establishing moments of 
truth around each digital interaction to build confidence. This means designing 
digital services focused on enhancing the consumer experience. Each positive 
experience builds people’s confidence in the competency of the public institu- 
tion. As levels of confidence grow, support can be generated for expanding the 
social license for public institutions to collect, manage, and use people’s personal 
data. 

With every advance in using personal digital data, new risks emerge, such as 
individual privacy, cybersecurity, algorithmic bias, ethics, and moral hazards from 
automated decision-making. Notwithstanding the risk, it remains possible for a 
service or exchange to occur based on the relative confidence at the point of 
interaction, even when there may be broader questions or doubts influencing the 
trusted relationship. For example, a person may have an overall low level of trust 
in government, but may still perform digital transactions and data exchanges with 
public entities. Convenience, ease of use, and value can be sufficient enough to 
generate confidence, while having little to no effect on a person’s assessment of 
trust. 

A further illustration of trust versus confidence is where people readily give up 
their personal information on commercial digital platforms, such as Facebook, 
Google, Uber, and Amazon, despite any evidence of a trust-based relationship. 
This occurs because people have confidence in receiving more value in return 
from a transaction or information exchange, despite the known risks. It is not 
uncommon, however, for the same people to be reluctant to share information 
with public entities, citing a lack of trust in the institutions of government. 
Therefore, the lesson for public entities is to not aim for the elusive goal of a 
trusted relationship with digital services. Instead, public agencies should focus on 
addressing the risks associated with each service interaction, with the intent to 
establish confidence in the service offering. 


84 Lee-Archer 


Action and Reaction 


The potential benefits of digital technology within public administration are 
expanding at an incredible rate, as are the risks. This class of risk is troubling for 
public administrators and citizens alike, and getting the balance of benefits and 
risks right is critical to success. The ever-increasing volume of citizen-related data 
coming into the hands of government is a source of great power, and with that 
comes great responsibility in ensuring it is managed correctly and used to create 
public value. 

In seeking better public policy outcomes, policymakers’ ability to take advan- 
tage of people’s personal digital data is contingent upon the level of confidence 
people have in their interactions with the institutions charged with managing the 
risks. Policymakers and administrators must follow a structured approach to 
managing the risks involved with digital initiatives and demonstrate transparency 
in the way risks are managed. One way to consider these emerging risks is in the 
context of Sir Isaac Newton’s Third Law of Motion, which states that for every 
action, there is an equal and opposite reaction (Westfall, 2020). Digital technol- 
ogy is enabling advances across every field of human endeavor. While these 
advances create forward motion, there is also the Third Law of Motion in play. 
With every forward change (progress) enabled by digital technology, there are 
opposing forces (risks). By adopting a structured approach to managing these risks, 
progress can be accelerated rather than held back. 

The technology industry has, for much of the 20th century, been able to 
sidestep many of the negative aspects of technology led social and economic dis- 
ruption. The risks have been accepted as inevitable consequences of digital led 
progress. However, as the pace of digital enabled change has accelerated in the 
first two decades of the 21st century, and as AI and machine learning have risen 
to prominence, public concerns are increasing over the risks associated with the 
use of personal data. Collectively these reactive forces are coalescing into sig- 
nificant inhibitors to collecting and managing personal digital data. 

In the context of Newton’s Third Law of Motion, the presence of reactive 
forces propel things forward. If society is to realize the potential of AI and 
machine learning in addressing public policy challenges, the risks need to be 
considered as the essential reactive force for enabling progress. To maintain for- 
ward momentum with the commitment to fair and equitable public policy, we 
need to lift our efforts in researching these reactive forces for the purpose of 
making them part of the solution, rather than immovable obstacles. 

Finding a balance between the contradictory forces of progress and the risks 
from personal digital data can also be considered through the lens of Chinese 
philosophy. The principle of Yin and Yang treats things as inseparable and con- 
tradictory opposites (Cartwright, 2018). These opposites attract and complement 
each other in reaching a point of harmony. Applying this principle to digital 
transformation, the contradictory opposites are technological change powered by 
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AI and machine learning, and managing the rising consequential risks in relation 
to ethics and privacy. Thus, there is much to be gained from data driven insight 
for public policymaking, providing there is a corresponding acknowledgement 
and commitment to researching the counter forces. 


Innovation, Regulation, and Oversight 


People generally expect digital services from public entities to be of a similar 
standard, level of convenience, and ease of use to commercial applications such 
as travel sites, online banking, and social media. However, while there are good 
examples of commercial digital services where people’s personal data are used in 
an ethical and responsible manner, there are many where this is not the case. As 
a result of some high-profile failures and data breaches in recent years, the 
technology industry has been called to account for how services are used and 
how they protect personal data. Given the amount of data within the digital 
education sector, there are even more reasons to challenge the technology 
industry and other public entities at risk of over-reaching in the use of people’s 
personal data. 

Around the world, there are examples of significant pushback against the 
misuse of data by technology companies, commercial entities, and governments. 
A major step in this direction is the European General Data Protection Regula- 
tion (GDPR) that came into effect in May 2018 (Proton Technologies AG, 
2020). The GDPR is the first of its kind, and it is intended to align privacy and 
data protection laws across Europe. It was developed to provide European citi- 
zens with a better understanding of how their personal information was being 
used. A key principle of this regulatory framework is the way in which personal 
information is collected and used with the informed consent of users. 

The introduction of regulation covering digital related activity is, at times, 
resisted by the technology industry, as there is a concern that over regulation may 
inhibit innovation. The technology industry’s capacity for delivering continual 
innovation has been a significant contributor to global economic and social 
development since the middle of the 20th century. Innovation occurs at the 
intersection of business insight and technology invention (Ruetsche, 2008). It is 
business insight that tempers and redirects the excitement and enthusiasm for the 
next big thing in technology (such as AI and machine learning) from being used 
for the wrong purpose or in an irresponsible manner. 

A structured and methodical approach to innovation is a key ingredient to 
success. Innovation is complemented by the discipline of public policy adminis- 
tration, rather than being incompatible with it. Having governance structures and 
processes to monitor performance and discontinuing non-performing programs 
are key components of an organizational innovation framework. Rather than 
thinking of innovation as simply digitizing everything and letting the technology 
industry control the innovation agenda through inventing new products and new 


86 Lee-Archer 


technology, making effective and responsible use of digital data provides business 
insight on where and what to transform. 

Positive storytelling can be an effective way to highlight public value creation 
arising from innovation initiatives. In the digital age, the power of positive 
storytelling remains just as relevant as it has through the ages. Organizations 
should be transparent and focus on generating positive stories centered on what 
works, highlighting where individual and public value are created from leveraging 
personal digital data (Accenture, 2019). To be credible, the stories need to be 
transparent on the identification of the risks and failures that may have occurred. 


Conclusion: Better Public Policy through Digital 


Public policy administrators seeking to accelerate the adoption of digital technology 
for value creation and reaching better outcomes for individuals and society are 
generally wary of new technology such as AI and machine learning. Education 
administrators have been even more cautious. This is understandable given the 
potentially compounding risks for children from the misuse of their personal data. 

Sustained and enduring transformation comes from a digitalization agenda 
where innovation is used to change how things are done. Digitalization needs to 
be prosecuted ahead of digitization initiatives, where attention is simply focused 
on automating the current state. This means, however, opening up personal data 
and using it in a transparent, ethical, and responsible manner. By taking deep 
dives to find where value is created for stakeholders and remaining proactive in 
calling out and mitigating the risks, organizations can lead a digital transformation 
agenda built on hope and optimism from AI and machine learning, rather than 
one clouded in fear and uncertainty. 

Focusing on people’s experience though digital moments of truth can lead to 
the establishment of confidence. Confidence cannot be taken for granted. It is 
time limited and related to the purpose of a digital interaction. It needs to be 
reconfirmed as each new digital service or offering is made and consent is sought 
to use personal digital data. As confidence builds and people increasingly engage 
in digital initiatives, previously provided consent for data sharing should be re- 
confirmed with individuals on a regular basis. 

The machines represented by the technologies of AI and machine learning, along 
with big data are already here. The role they play in public life is for public policy- 
makers, administrators, and human workers to decide in consultation with and 
through co-design with the people they serve. Following are four steps to guide lea- 
ders through this digital journey where machines augment human decision-making. 


1. Create the Environment for Innovation 


Tearing down barriers that inhibit innovation among teams is a key role of lea- 
ders. Build internal capability with appropriate external support (e.g., coaches) for 


Humans and Machines in Public Policy 87 


addressing data sharing, linkage, and protection issues. Establish a sound frame- 
work and methodology for innovation. Where appropriate, actively encourage 
teams to innovate around the complex issues starting with policy intent, through 
to program design and service delivery. Resist the temptation to focus solely on 
efficiency gains or to adopt technology simply because it is available. 


2. Nurture an Ethical Use Culture 


Senior leadership has a responsibility to embed the ethical use of digital data 
within the culture of the organization. This aspect alone is crucial to creating and 
maintaining people’s confidence in innovative digital initiatives. Much more than 
setting in place an ethics framework to guide leadership decision-making for 
projects, leaders must make cultural adjustments across the organization to ensure 
ethical considerations involving the use of personal data that are at the forefront 
of decision-making. 


3. Make Risk Your Friend 


The innovation mantras of “fail fast” and “don’t penalize risk takers” from the 
commercial sector do not have the same relevance in public administration. It is 
the role of leaders to set boundaries for risk appetite and risk tolerance, consistent 
with public accountability and transparency principles. There are instances in 
which innovation is required to address an issue or solve a problem, and failure 
may not be an option. Some risks, if they materialize, can be catastrophic for 
individuals and the organization. However, a fear of risks materializing should not 
be an excuse for inaction or a lack of progress. See risks as a positive counter- 
balancing force against the potential for over-reach and misuse of personal digital 
data. Explore how these counterforces can inspire innovation rather than inhibit 
it. 


4. Success in the Digital Realm Relies on Structure 


Digital innovation is not usually an overnight success. Innovation is dependent on 
structure to be successful. While digital transformation is couched in terms such as 
“agile development” and “minimum viable products,” successful outcomes are a 
function of well-defined scalable and flexible processes. Business and technology 
specialists collaborating through a co-design approach will achieve digitalization 
outcomes ahead of digitization. Organizations that consistently use business case 
analysis to assess each innovation initiative and conduct a return on investment 
analysis to evaluate implementation are generally more successful. 

Public policy has long advanced its many causes by realizing the benefits of digital 
technology adoption. This has largely been achieved with strong public support. 
While there were risks such as labor market disruptions from automation, these were 
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generally acceptable in terms of the overall gains in efficiency and effectiveness. 
Digital technology has now reached a point of advancement through AI, machine 
learning, and big data, where the risks are rising. These risks have the potential to 
significantly impact social stability as governments and corporate entities gain access 
to people’s personal digital data. The value proposition for addressing societies’ most 
complex social, economic, and environmental problems is strong. However, the 
downsides can be equally compelling. The way forward is highly dependent on how 
humans use the power of machines for the good of society. This demands a machine 
augmentation approach to human public policy decision-making. 
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AMAZON GO FOR EDUCATION? 


Artificial Intelligence, Disruption, and 
Intensification 


Kalervo N. Gulson, Andrew Murphie, and 
Kevin Witzenberger 


Introduction 


Amazon Go grocery stores are “a new kind of store with no checkout required” 
(Tillman, 2020, para. 4). The Go concept is the latest extension of the automated 
warehousing capabilities of Amazon. There are no shop assistants, they are 
replaced by an application (app) on customers’ phones. This app is powered by 
artificial intelligence (AI) that includes computer vision, deep learning algorithms, 
and sensor fusion technology (a system not dissimilar in general terms to that for 
self-driving cars; Tillman, 2020). The system registers products as they are picked 
up and as products leave the store. The app matches products automatically with 
customers. The system is constantly gathering data, not only about stock and 
process, but also about customer choices and behaviors in relation to products 
moment to moment. This kind of data is the so-called “oil” for future commer- 
cial AI development (Lee, 2018). 

Amazon Go’s use of AI does at least three things. First, it intensifies existing 
conditions of shopping by extending the existing automated checkout system 
where the customer scans the product. Second, this intensification produces new 
possibilities for data collection. Third, Amazon Go disrupts practices of face-to- 
face shopping at the same time as it disrupts the organization of labor with new 
kinds of efficiencies. As there is a complete removal of a physical checkout and 
related staff, customers learn to comport themselves — mentally, physically, and 
affectively — towards the system, with little or no access to human staff. 

In this chapter, we try to think about the intensifying and disruptive capacities 
of AI in education. We try to imagine the implications of a kind of Amazon Go 
for education, in which educational experiences of various kinds might be drawn 
on as required, without any intermediating assistants, and perhaps without the 


DOI: 10.4324/9781003045793-5 


Amazon Go for Education? 91 


need for the bricks and mortar of schools or even teachers themselves. Over the 
past 100 years, education institutions have largely assimilated technological 
developments, and have been relatively stable and very slow to change (Jonasson, 
2016; Koschmann & Kolodner, 1997). In this chapter we ask: Will AI follow a 
similar path in its introduction? What will be the roles of schools, universities, and 
the forms of organization that are attached to them as AI is introduced? It is not a 
given that education institutions will remain the same. Will we still need or have 
schools, universities, or the forms of organization attached to them in anything 
like the form we have them now? It is not a given that they will remain at all. 

The influential “historian of the future,” Yuval Noah Harari, suggests the 
important questions for education today concern whether we can predict what to 
teach in an AI world. He suggests that we cannot know what the world will look 
like in 20 or 30 years’ time. He is referring to not only, 


...the basics of geopolitics but what the job market would look like, what 
kind of skills people will need, what family structures will look like, what 
gender relations will look like. This means that for the first time in history we have 
no idea what to teach in schools. 

(quoted in de-Feytas- Tamura, 2018, n.p., our emphasis) 


This final point serves as an important caveat for this chapter. While we have 
good evidence relating to past and current trends in education and technology, 
the nature of applications of AI coming into education requires us to consider a 
very diverse set of possible future scenarios. This requires us to acknowledge 
inherent limitations on our capacities for prediction and planning, and to perhaps 
think in terms of preparedness in relation to the unexpected. 

This multiplicity of possible future scenarios is inherent in the term artificial 
intelligence, first coined by John McCarthy in 1955 to describe a heterogenous set 
of techniques and tasks, rather than a single “thing.” Definitions of AI follow suit. 
Boden (2016) provides a simple but fluid definition of AI as describing the field 
that seeks “to make computers do the sorts of things that minds can do” (p. 1). 
This involves efforts to produce “systems that think like humans, systems that act 
like humans, systems that think rationally, [and] systems that act rationally” 
(Russell & Norvig, 2016, pp. 2-3). Nonetheless, there is still no common 
understanding of what AI is or what it will become. Greg Corrado from Google 
Brain underscores that the focus in AI research on new challenges is: “not about 
what a machine ‘knows’ or ‘understands’ but what it ‘does’, and — more impor- 
tantly — what it doesn’t do yet’ (Lewis-Kraus, 2016, n.p.). Ironically, when suc- 
cessful, the development of new, specific AI capacities often results in the 
exclusion of these technologies from what is considered to be the domain of AI. 
Popular discussion about AI often conflates the term with “artificial general 
intelligence” (AGI) or “strong AI.” While strong AI continues to be the focus of 
AI research, it is unlikely that it will significantly impact education in the next 
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decade (Walsh, 2017). The focus of this chapter is on “weak AI” or intelligent 
agents that enable the automation of specific and narrow tasks using techniques 
primarily underpinned by machine learning.' 

The chapter proceeds as follows. We first outline our conceptual apparatus and 
methodological concerns, based on speculation and the challenge for educational 
research of examining the conditions that are yet to appear. The second part of 
the chapter provides our review of literature and our speculations about the 
intensification and disruptive possibilities of AI in education. We do not think 
that intensification and disruption in education are mutually exclusive, rather Al 
in education involves a series of “conjunctive syntheses” (Deleuze & Guattari, 
1983), which are not either/or but “and...and...and.”” Nonetheless, we separate 
these for heuristic purposes. We examine how the current organization of edu- 
cation may shape the ways any new AI technologies are adopted, including the 
influence of corporate interests and technological advances that will also shape 
this adoption. This view presumes that AI will intensify, complement, and sup- 
plement pre-existing practices and processes in education. We then look at pos- 
sible substantive disruptions to education, focusing on issues of provision and 
knowledge. We conclude the chapter by identifying issues we think require fur- 
ther examination. 


Conceptual and Methodological Note: Speculation as beyond 
Gray Literature 


“Prediction is very difficult, especially about the future.” 
— Niels Bohr? 


This chapter is based on secondary evidence. We reference academic work where 
it exists and have included few references to gray literature produced by organi- 
zations such as consultancy companies, or the rapidly growing body of popular 
books in this area. 

Additionally, we locate this chapter within the conceptual resources from cri- 
tical technology studies in education, new media studies, and science and tech- 
nology studies. Almost any new technology in the past has involved a great deal 
of speculation that has proved to be highly inaccurate, or at least trapped in lim- 
ited modes of thinking (Murphie & Potts, 2003, p. 8). It is worth also considering 
change from now will be simultaneously multiple — three of the largest changes 
are technical-media (AI and more), climate change, and broad social and political 
change. This will provide a “catastrophic multiplicity” (Murphie, 2018), in the 
sense of a dramatic turning away from how we think the future will unfold. It is 
difficult — likely distasteful, but necessary — to think through the demands of these 
changes together. In many ways, ours is truly a time without precedent. 

Such demands and changes lead to a pressing question: How can education 
research — with its alignment to evidence-based research practices — deal with the 
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consequences of technologies or ideas that do not yet exist? In this chapter, we 
want to put forward the use of speculative approaches as ways of thinking 
through the implications of the future. As an advocate of speculative methods 
within digital education research, Ross (2017) argues that “speculative approaches 
are aimed at envisioning or crafting futures or conditions which may not yet 
currently exist, to provoke new ways of thinking and bring particular ideas or 
issues into focus” (p. 215). Ross argues further that digital education research lacks 
an imaginative resource to take a strong position at the edges of educational 
change, where it is most needed (Ross, 2017). 

While this imaginary resource is perhaps lacking within education research, it is 
still present within the larger education sector among a range of stakeholders and 
policy actors that now include technology companies. Amazon is just one 
example of that. The company’s emerging technologies and company-funded 
research reports about their future implications are speculative devices and pro- 
positions themselves. However, in contrast to these primarily profit-driven spec- 
ulations, it is important for education research to “offer counter-visions which 
address, for example, issues of equality, diversity and social justice” (Ross, 2017, 
p. 218). 

Speculative research approaches are well-equipped to do this work, as they are 
complexity over certainty (MacLure, 2006), as a “speculative method is itself a 
proposition: neither the result of theories nor of practice, the rational nor the 
empirical” that “does not lead to vague ideas but starts from them to arrive at 
particular points of view” (Parisi, 2014, p. 240). Therefore, speculative approaches 
may be able to provide an imaginative resource for understanding the impact of 
AI on education. We attempt this in what follows. 


Intensification 


In this section, we are interested in the existing conditions of education that 
enable AI to either be already part of education or easily introduced. We focus on 
two kinds of conditions: one concerns education governance, the other compu- 
tation and education. 


Datafication and Data Infrastructures 


The present conditions of education governance allow AI to be seamlessly 
introduced, with key aspects including education being governed through datafication 
and administered via infrastructures. Datafication, as the process of translating things 
and events into quantities, is accompanied by the building of data infrastructures 
in education, which both enable forms of datafication and are themselves an 
outcome of the focus on digital data. Without wishing to rehash a now widely 
covered discussion, data infrastructures as sociotechnical forms, in which people, 
networks, algorithms, and computational capacities are all central to data 
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infrastructures in education (Anagnostopoulos et al., 2013; Hartong, 2018; Sellar, 
2015). 

Datafication and infrastructure can be seen as a continuation of the political 
rationality of measurement in governance including: i) education as a form of 
biopolitics and state management connected to the policy sciences in education; 
and, ii) “evidence-based” policymaking of the late 1990s that came into social 
policy areas as part of new forms of data-based policy provision. The introduction 
of more hardware and software — that is computing power — intensifies a com- 
putational political rationality in decision-making, one that is, therefore, far from 
antithetical to the policy areas of education. As such, computational decision- 
making based on techniques such as machine learning is easily introduced (Gulson 
& Webb, 2017; Williamson, 2016) as “the era of so-called ‘big data’ ushers in 
important new changes in educational policy, pedagogical practice, and institu- 
tional strategy” (Knox et al., 2020, p. 31). 

Students in elementary, secondary, and especially post-secondary education are 
increasingly likely to encounter automated basic communications. Student man- 
agement systems have already been established in multiple education sectors to 
collect and provide readable data, often in the form of data dashboards. This will 
likely continue, but will be combined with new forms of predictive analytics and 
data visualizations (Alexandru et al., 2015). AI will allow for the increased pro- 
cessing of data, in which decision-making done on the basis of extensive data 
from data warehouses will provide novel types of analytical insights. In higher 
education, learning management systems are already open to these developments, 
combining program and course administration with content provision, and can be 
further integrated with other systems, including student retention predictions, 
university administration and student records, ongoing program auditing, dis- 
ability services, and special consideration processes (Del Bonifro et al., 2020; 
Hoffait & Schyns, 2017). AI is likely to expand the category of educational data 
to include things such as simple metrics like attendance; student movement 
around a campus, such as visiting the library (or not); in-class engagement (such as 
attention, interaction, and motivation; Subramainan et al., 2016); and metrics on 
the affective aspects of learning, gathered in conjunction with adaptive learning 
(Avramides & du Boulay, 2009). 

The introduction of AI into education governance is part of what has been 
termed algorithmic governance, or predictive governance, as “a form of social 
ordering that relies on coordination between actors, is based on rules and incor- 
porates particularly complex computer-based epistemic procedures” (Katzenbach & 
Ulbricht, 2019, p. 2; see also Webb et al., 2020). In algorithmic governance all 
aspects of schooling — the affective, the interactionist, the pedagogical, curricula, 
and assessable — become translatable, if not translated, into numbers (as datafication), 
and subject to algorithmic logics. These logics are not outside of the existing modes 
of education, but AI introduces new forms of black boxing into decision-making 
that other kinds of data infrastructures have not yet indicated or enabled. 
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Pre-automation and the Structuring of Professional Obsolescence 


In predictions of professions under threat from automation, teachers have been 
identified as being at the lowest risk of all (Nedelkoska & Quintini, 2018), yet 
paradoxically as a profession, teachers been resigned to being made redundant 
(Selwyn, 2019). Either way, teaching roles are unlikely to stay the same. As 
Luckin (2018) notes: 


We live in times of financial stricture; there is therefore a risk that politicians, 
managers and decision-makers will be tempted by the inevitable enticements 
of a growing band of technology companies to believe that education and 
training can manage with fewer teachers and trainers, and that artificially 
intelligent tutors can be employed instead of human educators. 


(p. 138) 


While Luckin points out that replacing teachers with AI tutors would be erro- 
neous due to the required “human touch” of teaching, we can see that education 
provision, especially K-12, is increasingly focused on the “learning sciences,” 
which identify and replicate aspects of teaching and learning using AI capabilities. 
As AI based on pedagogical models is combined with “on-screen teaching” to 
underpin adaptive learning, the role of the teacher will change (du Boulay & 
Luckin, 2016). Some aspects of teaching will be automated that would previously 
seem the purview of human teachers, such as giving feedback on student collabora- 
tions (Floryan et al., 2010). 

While much of the discussion of teaching and AI has focused on the practice 
orientation of teacher replacement, we are also interested in the congruence of AI 
with some aspects of teaching (see also Selwyn, 2019). AI is very good at doing 
tasks that are part of highly structured hierarchies, and about identifying “a spe- 
cific range of programmed events (e.g., skills) and collections of programmed 
events (e.g., expertise) via which we now work and live” (Murphie, in press). 
The design of education, and we use that word “design” specifically, has long 
focused on standards and outcomes. While on the one hand this can be seen as 
encouraging faith in the profession, the converse is that it can mean that teaching, 
in a sense already “pre-automated” (Murphie, in press), is ripe for fuller automa- 
tions such as professional development that puts forwards toolkits.? This will in 
fact make it easier for full automation to occur, that is, a going beyond our 
behavioral automation to our full replacement. As Pasquale (2019) notes: 


The value of dialogue...in educational settings, should also be evident. There 
is wisdom in investing in human-focused... education teams, rather than 
dispatching these duties to smartphones and apps. Students... should resist 
pervasive appification because it is a short step to behaviorism. 


(p. 92) 
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But this type of functionalism is already part of the way schools are organized: 
ordering behaviors, and responses to the world, into basic elements and then reas- 
sembling into restricted and highly ordered diagrams and taxonomies allows things to 
more easily be “learnt” by machines, specific skill/expertise by skill/expertise. 

That is, much of the current system of organization of education is already 
based on the (pre)automation of learning and teaching. One significant part of 
this is that the roles of human teachers and students, and the experiences of 
learning and teaching, are highly constrained from above (through audit cultures, 
learning outcomes, etc.). This is based largely on quite old thinking about orga- 
nizations (Bloom et al., 1956). This could be a problem. For one thing, it means 
we are currently educating students and professionalizing teachers into a mode of 
engagement with the world which, by inducting them into pre-automated beha- 
viors, actually positions them as workers to be the more replaceable by full auto- 
mation. If education is organized according to these principles, down to such 
questions as “what skills do we need to meet the future of work?,” we are essen- 
tially meeting automation by speeding it up. We risk making students and teachers 
more replaceable by educating them, via forms of organization and modes of 
thinking and behaving that induce the lifeless lives that are most easily replaced. For 
AI would perform these lifeless functions more efficiently, at much lower, or no 
cost, with a flatter structure. AI would also deliver such functions much more 
flexibly, more integrated into networked infrastructures, and with a potential for 
much faster adaptation to world changes. If we already assemble ourselves to 
operate algorithmically, why would an algorithm not be able to replace us? 


Disruption 


This section outlines three areas where we think there are interesting, if some- 
what confronting, possibilities of disruption with the introduction of AI. We try 
to move past the common narrative of challenges to employment and skills to 
focus on the transformation of provision and knowledge. 


Human-machine Learning 


If the aim of AI research has often been assumed to model minds (Boden, 2016), 
this also has implications for our conception of human learning and our under- 
standing of human brain function. Our conceptions of machine intelligence, 
human minds, and learning (human and non-human) are increasingly inter- 
connected, with changes in one domain having potential impacts on thinking 
about the others. It is important to note that AI itself also needs to learn. Com- 
puter-based intelligent agents, particularly those based on artificial neural net- 
works, are “machine learners” (Mackenzie, 2017) that are trained on patterns in 
big data sets and reinforce these patterns in their actions. While this is quite a 
straightforward equivalence of learning, based on an anthropomorphic attribution 
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of learning, we could also entertain the possibility that in an AI world, our con- 
ception of learning and education could change, as could our perception of the 
world and ourselves through our engagement with AI embedded in new media. 

Additionally, some unusual effects arise if AI is, rightly or wrongly, often strongly 
tied to concepts of human learning, even to the way the functioning of the human 
brain is conceived (so that when the way that neurons work is reconceived, this is 
often hailed as leading to a possible breakthrough in machine learning). In short, AI 
and education overlap because learning itself, and core assumptions related to learn- 
ing, are central to both (Luckin, 2018). This poses a problem for trying to plan 
education, as when one aspect of the following equivalence changes, the others are 
often conceived differently. 


What is a machine <—> AI <—> learning <——-> what is mind? <———> 
what is human? 


That is, “learning” might change through notions that are carried over from other 
fields. AI itself, somewhat ironically, could propel education away from the 1950s 
model of cognitivism still common in educational assumptions about learning. That 
is, learning narrowly defined via data storage and retrieval for both human and 
machine. More recent versions of AI include new models of mind somewhat differ- 
ent to the old model provided by a 1950s computer, in favor of the importation of 
models of learning drawn from more open network computing and neural networks. 
Events of learning are seen to “emerge” stochastically across these networks, as 
opposed to the more structured module-based “cognitivist model” in “good old- 
fashioned AT” that still informs much of cognitivist educational thinking. The ques- 
tion arises if thinking is a matter of the logical processing of symbols based on learnt 
procedures and stored information (e.g., cognitivism), or is it a matter of many itera- 
tions of high-level pattern matching and recognition in response to a given situation 
(e.g., facial recognition; Mackenzie, 2017)? 

At the same time, we might ask whether AI and human thinking have to be 
conceived in the same way, or whether this is not in itself a deeper outmoded 
assumption, one found in AI development itself often enough, but one also found 
in many educational assumptions, including those underpinning educational 
organizations. AI is assumed to replicate human-like skills, which is sometimes 
involved. However, why does AI have to be human-like, and why indeed does 
AI therefore have to be conceived as replacing humans? Music artists such as 
Holly Herndon and German duo Grandbrothers have for years worked across the 
differences between humans and machines.* 


Education Technology Platforms as Education Providers 


AI and related technologies enable new platforms and technical innovations that 
can disrupt entire industries and fields (Eriksson et al., 2019). For example, mp3 
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and file compression more generally disrupted not only the music, film, and book 
industries, but entire habits of engagement with music, film, television, and 
reading. The Internet and social media disrupted the entire journalism industry. 

As noted, AI is easily introduced through data infrastructures in education, 
those already existing or emerging sociotechnical apparatus that allow for the 
integration and intervention of data into the life of schools and systems. Platforms 
extend data infrastructures, and as these are primarily proprietary, also add a new 
type of AI supported corporatization to education (van Dijck et al., 2018). Plat- 
forms create new connections in a “global-scale arrangement of planetary-scale 
computing” (Bratton, 2016, p. 44), and are an amalgam of technical aspects, 
computation, and new forms of sociality in which a platform is a “standards-based 
technical-economic system” (p. 141—142). As Srnicek (2017) notes, “platforms 
are digital infrastructures that enable two or more groups to interact. They 
therefore position themselves as intermediaries that bring together different users” 
(p. 43). 

The growth of platforms is concurrent with the extension of AI applications 
into education. That is, there is a dual growth of corporate platforms and AI 
within education, as education providers are supported by AI, but also provide 
forms of entry to AI applications (Williamson, 2019). AI itself is a kind of non- 
platform, in that it can move across different platforms or experiences. One side 
to this is that when AI and data connect, there is a massive increase in the ease of 
distribution across the networks involved. This includes a potential distribution of 
learned functions, that is, learned by AI agents, and of educational events and 
content. It can occur across multi-scalar events of networking, from micro- 
instances of a learning behavior to global circulations. The result is that all systems 
in a network can almost instantaneously learn what another system has learned, a 
process that Walsh (2017) calls “co-learning” (p. 182).° Platforms (such as 
Moodle) often deployed in universities cannot hope to match many corporate 
deployments or the potential of AI. In part because they have nothing like the 
kind of technical capability or support needed, but also because they are plugged 
into forms of organization that run counter to ease of use and good user 
experience). 

What is enabled through platforms, which perhaps gives corporate platforms an 
advantage, is the incorporation of automation through application programming 
interfaces (APIs). Mackenzie (2019) posits that APIs become more important as 
parts of platforms include AI, and move from “a mode of programmability 
focused on linking systems to a mode of programmability — AI or machine 
learning — centered on prediction” (p. 1990). APIs allow for the access of data, 
processing, new third-party developers and so forth. It means that within one 
platform there is a new dispersed spatiality to how education is delivered, pro- 
viding a complex web of potential connections among disparate educational 
developers and providers. Understanding this allows us to understand more fully 
what platforms are enabling within education, when “platforms act: they perform 
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specific operations (and others not); convey particular messages (and others not); 
draw some things in (and others out); make choices in what can appear (and what 
not); and how it is organized (or not)” (Decuypere & Landri, 2020, p. 4). Some of 
these platforms are specifically educational, such as Google Classroom and higher 
education platforms like U-Multirank (Decuypere & Landri, 2020). These platforms 
are both explicitly pedagogical (e.g., Google Classroom) or introduced as governance 
platforms, such as business intelligence systems (Sellar & Gulson, 2019). 


Literacies and Skills in a “Third Media Revolution” 


Our final section on disruption explores the ways that any response to new 
technologies requires epistemological transformations. Within education, these are 
broken down to questions of literacies and skills, which are key questions 
informing the organization of education itself. For example, in writing as a uni- 
versity president on the need for higher education to respond to the challenges of 
AI, Aoun (2017) calls for “new literacies” that comprise data literacy, technolo- 
gical literacy, and human literacy, arguing that people need digital not just ana- 
logue tools: 


they need data literacy to read, analyze, and use these ever-rising tides of 
information. Technological literacy gives them a grounding in coding and 
engineering principles, so they know how their machines tick. Lastly, human 
literacy teaches them humanities, communication, and design, allowing them 
to function in the human milieu. 


(p. xix) 


This kind of multi-modal approach is common and is built into calls for 21st 
century skills in K-12 education, such as creativity, critical thinking, commu- 
nication, collaboration, information literacy, and self-direction (Chung, 2017). 
These types of skills can be contrasted with, for example, a focus on coding that 
has been somewhat misdirected, as it is likely that some forms of AI will not need 
to be coded. As Luckin (2018) posits, building future AI systems “will be about 
clever design and comparatively less will be about writing computer code. To 
some extent at least our future AI systems will be able to code parts of them- 
selves” (p. 134). 

Yet, much of our current organization, social life, and indeed economy, are 
based on the literacy and skills needs within what Murphie (2018) calls a first and 
a second media revolution. The first media revolution involved the development 
of modes of expression more open to abstraction, especially writing. This changed 
the world in dramatic ways, from religions of the book to Western philosophy’s 
take up of writing as a record of forms of reflection, to commerce, and capital’s 
new ability to keep accounts. The second media revolution, from the printing 
press through to the present environment of social media and texting and, from 
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some perspectives, computing itself, involved the copying and wide distribution 
of representations (words, picture, films, but also texts on mobiles, posts on social 
media pages, and code). This also changed the world dramatically and famously, 
in a scientific and technical revolution enabled by the distribution of knowledge, 
the rise of democracy, publics, and the nation-state, enabled in part by news- 
papers and national literatures, and much more (Eisenstein, 1979). 

Much of the organization of education, the current form of which was very 
much developed to enhance literacies in writing (for example, rows of desks, and 
courses and programs with relatively fixed structures, that consist of representa- 
tions to be “processed”). This reflects a long history of a certain kind of economy 
and social life, based on the circulation and engagement of representations; this is 
also how we usually think, too narrowly perhaps, of media and communications. 
Even more narrowly, much educational organization is conceived in the very 
particular second media revolution terms of the 1950s. In this, the “information” 
models (such as cognitive psychology) that inform much of the understanding 
and organization of education are based on the idea that knowledge acquisition 
involves the reception, storage, and possible retrieval when needed, of informa- 
tion (whether as “knowledge” or as “skills”), via symbolic processing and as 
symbolic representations. 

Yet AI is part of a transition into a “third media revolution” (Murphie, 2018) 
along with: interactive technologies; virtual, augmented, and mixed realities in 
their transformation of perception; hyper-networking, global networks of sensors, 
and the Internet of Things (IoT); genetics and optogenetics, drones, robotics, and 
bots; and algorithms in financial trading. The third media revolution uses or 
intensifies elements of the first and second media revolutions. Yet it does so in a 
manner that changes the game significantly. Writing and representation lose their 
“primary” status in events. 

In the third media revolution, media and communications are no longer con- 
tent to just represent the world. Rather, third media revolution technologies aim 
to actively fold media and the world into each other. Media and the world 
transform each other via third media revolution technologies, often quite dra- 
matically. These technologies read the world in order to act within it; think a 
drone attack or IoT technology that adjusts temperature and lighting. 

AI for example is not only a matter of writing and representation but a media 
technology that can, for example in self-driving vehicles, respond to the world in 
real time. Indeed, the whole point of AI is to act in the world and to enable 
other technologies to act in the world (such as robotics). In the third media 
revolution, media find new forms and many new niches in the world, or order to 
modulate and be modulated by the world. It is this transformation of the world 
that is in part driven by AI. It still requires older literacies, but also requires new 
literacies. In fact, beyond “literacy” per se, it requires new understandings of 
activity, what it is, and what it is to be active in a world in which world and 
technologies so intensely modulate each other. Only one aspect of this might be 
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an entirely new way to conceive of “active learning” as an educational potential 
within these modulating folds of world and media. 


Avoiding Technological Somnambulism: Enlarging Our 
Considerations of Al in Education 


Like other areas of social and political life, decisions about the uses of AI in 
education need to be enlarged beyond considerations of what is technically pos- 
sible at this point in time. Winner proposes that the problem is not so much 
technological determinism, but rather “what might be called technological som- 
nambulism — how we so willingly sleepwalk through the process of reconstituting 
the conditions of human existence” (Winner, 1983, p. 254)°. To attempt to avoid 
this technological somnambulism around AI in education, in this conclusion we 
try to identify issues that we think need to be further considered “to provoke 
new ways of thinking and bring particular ideas or issues into focus” (Ross, 2017, 
p. 215). 

First, much of what we have covered as intensification and disruption assumes 
a continuation of the “place” of schooling, where schools are likely to remain 
important as “locally accountable organizations, committed to building viable and 
sustainable futures for everyone in their communities” (Facer, 2011, p. x). 
However, AI could, in tandem with hypernetworks and augmented reality, 
fragment, reassemble, and distribute education very differently, in a manner much 
more situationally sensitive. Due to the capacity to deliver not only content, but 
different pedagogical modes and assessments, it is likely that there will be a focus 
on mobile learning connected to new forms of collaboration (Woolf et al., 2013, 
p. 72-73). Additionally, the places of schooling may become networked beyond 
our current experiences. One would always have access to the world, and to 
specific, situationally required, learning experiences and knowledge. This could 
include education connected to new forms of virtual reality or augmented reality 
glasses (Southgate, 2020). Other networks may build on existing technology, such 
as Apple’s AirPods (Brandt, 2016), which could be always worn in the ears and 
always connected not only to the Internet as we know it, but also to satellite 
systems, hypernetworks, the IoT in an updated version of William Gibson’s 
cyberpunk world of Neuromancer (Gibson, 1984). 

Second, education might be offered by a range of different providers. Where 
will schools and universities position themselves in the midst of this and why 
would students enroll in them? 

Challenges will include making decisions about equity of provision, and reg- 
ulations that will be necessary for delivery and for credentialing, in whatever 
forms these take, if indeed the latter survives in anything like its present form. 
Many of these issues will be able to draw on existing regulations concerning 
home-schooling and online forms of schooling, such as online charter schools 
(Barbour & Reeves, 2009). What might be different with AI is that we will need 
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to consider what to do about regulating technology companies (Pasquale, 2020) 
not only in education, but also companies such as Google that are often located 
both inside and outside education. 

Third, as an alternative to this, AI could disrupt the whole current culture of 
the proprietary platform. New technologies, including blockchain or post- 
blockchain technologies, could decentralize organizations away from large scale 
platforms — a vision that is part of the speculative fiction of the post-Internet 
world, such as the novel Infinite Detail (Maughan, 2019). While the focus of 
much work on AI in education is on the role of corporate entities, perhaps edu- 
cation will become a kind of trade among equal members of the community. It 
could lose its hierarchical forms of organization. In partnership with blockchain 
and other technologies, AI could assemble and communicate ongoing accredita- 
tion, in an accreditation that is much more nuanced, detailed, and responsive to 
the situations of workers and employers. While education could become like 
Amazon Go and become captured by platform capitalism, it could also be thor- 
oughly democratized in platform cooperativism (Scholz & Schneider, 2016). 
Alternative forms of provision and alternative providers, from corporations to 
platforms allowing a kind of education marketplace to community or cooperative 
organizations, could drive down costs to near zero marginal cost. If so, what will 
this do to the revenue models of schools and universities, or government budgets 
providing education? Why would students pay for a certain kind of experience at 
schools or universities when they could get something that is better, cheaper, and 
more flexible, elsewhere? 

None of this is a given (Walsh, 2017). However, all this makes it more dan- 
gerous to imagine the wrong world, based on poor or outmoded models, 
including within the organization of education. The most likely eventual educa- 
tional world involving AI would be a complex world in which some things Al 
does are intrusions on current human activity, others might be totally alien to the 
human — machine but not human interpretable — and yet others somewhat alien but 
complementary to the human. This is before we get to the interaction of AI and 
education with other domains such as climate change (Doctorow, 2020).’ 
Therefore, we think a particular ethos is needed here — to be open to the co- 
adaptation of humans and machines, both involved in learning, and both learning 
to work and to learn together. As Luckin (2017) contends, “we need to develop 
a culture of problem specification that encourages people to unpack educational 
problems, so that solutions that benefit from the symbiosis of AI and human 
intelligence can be developed” (p. 124). 

All this makes speculation about the future more difficult, but all the more 
necessary, if we are to understand speculation not as prediction, but as 
exploration of a multiplicity of potentials events, given the current multiplicity 
of tendencies at work. AI and related technologies could not only intensify and 
disrupt the current provision of education, but entire habits of engaging with 
education itself. 
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Notes 


1 Aspects of the above sections are reworked from the following report: Gulson et al. 
(2018). Education, work and Australian society: A review of research literature and policy 
recommendations. University of New South Wales/Gonski Institute for Education. 
Thanks to Sam Sellar and Simon Taylor for their contributions that are used in this 
chapter. 

2 Variously attributed to Niels Bohr, an old Danish proverb, and so on. Only proving 
that it is just as difficult to ascertain the past as it is to govern the future. 

3 Examples of toolkits include focusing on toolkits as ways of organizations (schooling) 
being “agile” and “adaptive” (see Evidence for Learning, n.d.). 

4 Creative applications are well advanced in this area such as Holly Herndon (see Holly- 
herndon.com, n.d.) and the German duo Grandbrothers (see Peake, 2017). 

5 Co-learning is when “an agent in a collective group learns EITHER directly for 
themselves OR indirectly from another agent” (Walsh, 2017, p. 182). 

6 Thanks to Simon Taylor for alerting us to Winner’s work. 

7 Speculative futures writer Cory Doctorow (2020), who is also an “AI skeptic” as well as 
“an automation-employment-crisis skeptic” (para. 3), recently suggested that full 
employment was likely for at least 200-300 years due to the labor-intensive tasks that 
will occur due to remediating climate change. 
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PEARSON'S DIGITAL 
TRANSFORMATION AND THE 
DISRUPTION OF PUBLIC EDUCATION 


Anna Hogan and Sam Sellar 


Introduction 


Pearson characterizes itself as the “world’s learning company” (Pearson Central 
Europe, 2020, para. 2), and it has a presence in more than 70 markets globally. 
The company has undergone significant restructuring over recent years, moving 
from an Anglo-American media holding company to a globally integrated edu- 
cation services company. In 2018, revenue from digital and digitally-enabled 
products and services accounted for 62% of sales, illustrating the extent of Pear- 
son’s digital transformation to date. In this chapter, we critically analyze Pearson’s 
current key strategic priorities: (1) growing market share through digital trans- 
formation and (2) investing in structural growth markets (e.g., online provision 
and computer-based testing; Pearson, 2020a). Pearson is following the corporate 
vision of many technology companies that emphasize digital disruption to existing 
markets. Pearson’s strategy is not informed by a coherent set of educational or 
social principles, but rather by the aim to protect and grow shareholder value. 
Pearson’s corporate success depends, in part, on disrupting the public provision of 
education. 

Pearson aims to lead the next generation of teaching and learning through the 
development of digital learning platforms, including the use of artificial intelli- 
gence (AI) in education (AIEd; Luckin et al., 2016). The company is also piloting 
AI technologies that it hopes will provide automated, real-time feedback to stu- 
dents in the form of a virtual tutor, much like Apple’s Siri or Amazon’s Alexa. 
This technology will be integrated into a single platform — Pearson Realize — that 
will deliver its digital services in an individually personalized way (Olson, 2018). 
Pearson envisions direct and lifelong relationships with customers to whom it will 
provide virtual schooling, professional certifications, assessments, and other digital 
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services. The potential impacts of these developments on public education 
include: changes to funding and resource distribution; reductions in the need for, 
and benefits of, teachers; and the private accumulation of large volumes of stu- 
dent data, which creates a range of new risks. 

In this chapter, we critically analyze Pearson’s strategic priorities and broader 
corporate strategy. We begin by reviewing the digital transformation of the 
Global Education Industry (GEI) and discuss why Pearson and many other edu- 
businesses are focusing on “digital first approaches” to learning. We then conduct 
a scoping review of key reports commissioned by Pearson to show that it is 
steering contexts of policy influence in ways that align with its corporate strategy of 
digital transformation. We discuss Pearson’s vision of the wide-scale adoption of 
personalized learning, which entails disruptive changes to teaching, curriculum, and 
assessment, and the organization and function of schools and universities. We con- 
clude by raising two main causes for concern in relation to Pearson’s vision and its 
implications for the integrity and sustainability of public education globally: firstly, 
the privatization of data and data infrastructures, through what Harvey (2004) 
describes as “accumulation by dispossession”; and secondly, and the transformation of 
the teaching profession and the broader social purposes of education. 


The Digital Transformation of the GEI 


There has been an explosion of edu-businesses seeking to capitalize on burgeon- 
ing education markets over the past two decades. A growing body of research has 
described the rise of the GEI and its key characteristics, including: 


public-private partnerships (Steiner-Khamsi & Draxler, 2018); 

the contracting of assessment services (Hogan, 2016); 

the private management of public schools (charter schools, academies, and 
free schools; Fabricant & Fine, 2015); 

e the outsourcing of schooling to the private sector (low-fee private schools in 
Sub-Saharan Africa, South America, India, and parts of Asia; Srivastava, 
2016); and 

e the multitude of products and services on sale from the private sector to 
support the day-to-day practice of schools (for example, curriculum, learning 
resources, digital technologies, teacher professional learning, administrative 
tools; Molnar, 2018). 


Major capital investment is now being concentrated on the digital transforma- 
tion of education by major technology companies and venture capital investment 
firms (Williamson, 2017a). There is now unprecedented optimism for the 
potential of education technology to transform schooling, and in 2017 more than 
US$9.5 billion in investments were made to education technology (EdTech) 
companies (Morrison et al., 2019). 
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According to Metaari’s (2018) commercial analysis reports, there are seven 
types of learning technology products: Al-based learning; mixed reality learning 
(i.e., the integration of augmented reality and virtual reality); game-based learn- 
ing; mobile learning; cognitive learning (i.e., behavior modification products); 
location-based learning (i.e., learning triggered at geotagged physical locations); 
and educational robots (i.e., robots to teach programming and related skills). All 
of these products have a growth rate of more than 14% per year, with educational 
robots having the highest growth rate (30.5%) and mobile learning the lowest 
(14.4%). Interestingly, mobile learning revenues are the highest of all product 
types with global revenues set to double to $26 billion by 2023 (Metaari, 2018). 
Metaari suggests there are six market segments that buy EdTech products: con- 
sumers, Pre-K—12, higher education, corporations/businesses, federal govern- 
ments, and state/local governments. The factors driving the demand for these 
products are different in each buyer segment; for example, in the pre-K—12 
market, Al-based learning is experiencing a rapid uptake due to the promise of AI 
being able to achieve true personalized learning for the first time (Metaari, 2018). 

The integration of AI into digital learning is currently the “holy grail” of 
EdTech, and one that is having a profound impact on the GEI. For example, the 
Watson Education Team at IBM has announced various partnerships with Pear- 
son, Apple, Houghton Mifflin Harcourt, Scholastic, Elmodo, and others to bring 
AIEd to mainstream schooling. IBM (2019) argues that Watson is a cognitive 
technology that can “think like a human” by understanding, analyzing, and 
interpreting data; using machine learning to grow subject matter expertise in apps 
and systems; provide personalized recommendations based on understanding a 
user’s personality, tone, and emotions; and interact using chat bots that engage in 
dialogue. Collaborations that combine education content with Watson’s AI cap- 
abilities can generate various personalized learning products that can be sold to 
learning institutions. For example, Watson is used as a personalized, virtual, one- 
on-one tutor within some Pearson courseware products (Pearson, 2016). 

Pearson is a particularly interesting case to explore the digital transformation of 
education. The company has an extensive portfolio of digital products and services, 
including courseware, assessment, and learning platforms that encompass many 
EdTech product types. Pearson previously specialized in education textbooks, but the 
company has shifted towards a digital learning model and providing more direct-to- 
client services. Initially, its profitability suffered under this model with former Chief 
Executive Officer John Fallon describing the transition from analogue to digital as 
being a painful period for Pearson. However, the potential for future profitability has 
driven Pearson to re-focus its business priorities over recent years by simplifying its 
portfolio to invest in its digital transformation. In 2016, Pearson invested £700m+ to 
drive the digital transformation of its portfolio of products and services, with the same 
financial outlay being made again in 2017 and 2018. The 2018 annual report 
announced a return to profit, despite a decline in sales, and an ongoing commitment to 
the recovery of the business through digital transformation (Pearson, 2018). However, 
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Pearson has continued to struggle and has issued a series of profit warnings in recent 
years, including in response to COVID-19 disruptions (Pearson, 2020b). 

Studies of the development of Pearson’s education business have criticized its 
prioritizing of shareholder value over the interests of students, teachers, schools, 
and communities (Ball, 2012; Ball et al., 2017; Hogan et al., 2015; 2016; Hursh, 
2015; Riep, 2017a; 2017b; Srivastava, 2016; Williamson, 2016). Of course, 
companies must prioritize shareholder interests, and Pearson has also actively 
pursued its corporate social responsibility, but its ambitions and growing influence 
in education risk creating an imbalance between the private value and public 
benefits of education. In this chapter, we explore a range of new and emerging 
concerns associated with the digital shift to education. For example, Williamson 
(2017b) raises concerns about patents and intellectual property in this new 
environment. He argues that corporations are using data infrastructures to mine 
and analyze big data in order to become powerful actors in educational research, 
knowledge production, and theory development. He states: 


these are by no means neutral, value-free or atheoretical approaches. They 
are the product of academic and commercial actors animated by a specific 
imaginary of a desirable future of educational research and development, 
working within a field of power defined by its economic, social and cultural 
capital, whose practices reflect a particular data scientific style of thinking that 
views learning in scientific terms as quantifiable, measurable, actionable and 
therefore optimizable. 

(Williamson, 2017b, p. 119) 


Corporations can increasingly shape the pedagogic practices of teachers and the 
learning processes of students, and we argue that Pearson has also been working to 
steer contexts of policy influence in ways that align with its digital transformation 
strategy. 


Methods 


The chapter presents a scoping review of Pearson’s commissioned reports in 
education. Specifically, we identify the reports published by Pearson through its 
“Open Ideas” forum between 2012 and 2018. Mays et al.’s (2001) definition of a 
scoping review is a process that aims to “map rapidly the key concepts under- 
pinning a research area and the main sources and types of evidence available” (p. 
194, emphasis in the original, in Arksey & O’Malley, 2005, p. 21). A scoping 
review can be conceived as a method in its own right, and can be undertaken to 
examine the extent, range, and nature of research activity, rather than to describe 
research findings in detail like a systematic review. Our purpose in undertaking a 
scoping study of Pearson’s commissioned research is to ask what research narrative 
Pearson is creating around the digital transformation of education? 


Pearson’s Digital Transformation 111 


Arksey and O’Malley (2005) argue there is no definitive procedure to follow 
for a scoping review, but it is important to identify the research question, clearly 
define the inclusion/exclusion criteria of relevant studies, chart the data, and 
collate the findings. Only research papers published by Pearson through its 
“Open Ideas” forum were included in this scoping review. Using this criterion, 
we identified 16 research papers published between 2014 and 2018. Of these, six 
papers were excluded because they did not focus explicitly on digital transfor- 
mation: one paper focused on behavioral sciences, one on teacher expertise, one 
on the “effect size” of popular education reform policies, two on effective peda- 
gogies, and one on the development of business efficacy in education. The 
remaining 10 papers (over 60% of Pearson’s commissioned research reports) 
directly focused on digital learning in K-12 schooling and higher education. 

Charting the data involves identifying key items of information from the 
research reports being reviewed using a descriptive-analytical method (Arksey & 
O’Malley, 2005). Our coding scheme included the following information about 
each paper: year published, author, title, research focus, nature of analysis, and 
key outcomes/recommendations. The intention of this scoping review is not to 
present a view regarding the “weight” of evidence that Pearson provides in 
relation to digital learning in education. Our aim is not to assess the quality of 
evidence in research reports, and consequently we cannot comment on the 
robustness or generalizability of findings. Rather, we aim to identify the domi- 
nant areas of research into digital learning published by Pearson and the key 
recommendations. 


Charting the Promotion of Digital Learning in Pearson’s 
“Open Ideas” Forum 


Pearson states that its goal is “to help people make progress in their lives through 
learning.” They further note that “Open Ideas is one of the ways in which we do 
this. We work with some of the best minds in education — from teachers and 
technologists, to researchers and big thinkers — to bring their independent ideas 
and insights to a wider audience” (Luckin et al., 2016, p. 3). 

The section below outlines the key information from the 10 reports included 
in this scoping review. It is clear, that since 2014, Pearson has focused on com- 
missioning research that highlights the potential of technology to transform edu- 
cation. This began with the call to recognize the ubiquitous nature of technology 
and learning today, ensuring that all schools have the hardware, infrastructure, 
and staff capacities required for students to work with and benefit from digital 
tools and resources. The three reports published in 2014 all argue that technolo- 
gical advances are giving rise to a “next generation” of learning in schools. Each 
report details the “explosion” in student data that can be captured and analyzed 
through online learning, which can lead to more personalized and optimized 
learning for students, but involves a paradigm shift in assessment. Hill and Barber 
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(2014) and DiCerbo and Behrens (2014) argue that assessment should become 
integrated within personalized learning systems, providing continuous and 
immediate feedback to students on their everyday activities. 

The focus on personalized learning is expanded in two reports published in 
2016. These reports focus on adaptive learning (EdSurge, 2016) and AIEd 
(Luckin et al., 2016). EdSurge defines adaptive learning as tools that might have 
adaptive content, adaptive assessments, or adaptive sequences (for example, tools 
that continually collect and analyze student data to automatically change what a 
student sees next, from the order of skills to the type of content). They argue that 
the most popular adaptive tools currently in the K-12 market are those that focus 
on adaptive content for mathematics education. The EdSurge (2016) report sug- 
gests that the most successful adaptive tools will be “like an ever-present teaching 
assistant, supporting students with instruction while capturing information that is 
hard for teachers to regularly collect” (p. 30). Similarly, Luckin et al. (2016) argue 
that AIEd can be used to create 


the possibility of learning that is more personalized, flexible, inclusive, and 
engaging. It can provide teachers and learners with the tools that allow us to 
respond not only to what is being learnt, but also to how it is being learnt, 
and how the student feels. 


(p. 11) 


The Pearson report published in 2017 moved from a discussion of the need for 
digital learning in schools to examine how schools and educators can ensure the 
efficacy of EdTech products they purchase. However, Means et al. (2017) explain 
that this is not as simple as asking “what works?,” but requires understanding that 
it is difficult to “disentangle the impact of a learning technology from the effec- 
tiveness of the overall instructional system in which it’s embedded” (p. 6). Indeed, 
their report recommends that successful implementation of software would 
include “instructor” professional development about the new practices expected 
of educators. The efficacy of a learning technology product cannot be separated 
from the interactions between students, instructors, and learning activities in par- 
ticular contexts, and this means that organizations need to support teachers to use 
data about what is working and continually change their practice to support these 
ends (Means et al., 2017). 

Finally, the reports published in 2018 expand the focus of digital learning from 
K-12 schooling to focus on higher education. The theme that emerges from these 
reports is that universities need to offer more flexible learning pathways for students 
and further embrace the potential of technology to provide enhanced personalized 
learning opportunities for students. The underpinning rationale is that new, more 
dynamic modes of delivery will improve student access and completion by instan- 
tiating a consumer model of higher education that enables individual students to 
want to engage with their learning in ways that best suit them. 
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A theme across all of the reports is that teachers need to be better supported in 
classrooms and that this is possible through the rise of personalized learning, 
adaptive tools, and AIEd. The teacher is re-imagined as a classroom facilitator, 
rather than an imparter of knowledge (Hill & Barber, 2014). Algorithms, not 
teachers, will set curriculum, assess learning, and report on outcomes. In fact, a 
common recommendation across the reports is that teachers need additional 
professional training to ensure that technology is implemented in ways that best 
support student outcomes (see Fullan & Langworthy, 2014; Hill & Barber, 2014; 
Means et al., 2017). 


The 10 Reports from Pearson’s “Open Ideas” Forum 


1. (2014) Michael Fullan & Maria Langworthy: A rich seam: How new pedagogies 
find deep learning 

Focus: New pedagogies, enabled and accelerated by pervasive digital tools and 
resources, measure and support deep learning at all levels of the education system. 

Nature of Analysis: Report informed by interviews with “practitioners of 
new pedagogies and deep learning” including students, teachers, school leaders, 
and education experts that provide background to “deep learning” and evidence 
of this type of learning in practice. 

Key Findings or Arguments: Technology is ubiquitous in schools today. This 
requires every student to have a device, digital resources that align with curriculum, 
technology training for staff, high-speed Internet access, integrated assessment and 
monitoring programs, reporting mechanisms to allow frequent learning from work, 
communication with parents, infrastructure that includes security and privacy pro- 
tections, support and maintenance of equipment, and a digital citizenship policy. 

2. (2014) John Behrens & Kristen DiCerbo: Impacts of the digital ocean on 
education 

Focus: Big data can inform decisions about learning through systems that 
continually assess and provide feedback on student progress. 

Nature of Analysis: Opinion piece informed by literature arguing for new 
conceptualizations of learning to embrace a digital-first, data-first world. 

Key Findings or Arguments: Call for a paradigm shift in assessment that uses 
technological advances to collect data, identify patterns, provide immediate feed- 
back, generate learner profiles, and recommend leaning activities, e.g., persona- 
lized learning. 

3. (2014) Michael Barber & Peter Hill: Preparing for a renaissance in assessment 

Focus: Digital technologies and personalized learning systems are transforming 
student assessment. 

Nature of Analysis: Opinion piece informed by literature arguing for assess- 
ment reform through personalized learning systems. 

Key Findings or Arguments: Assessment should be integrated into “next 
generation learning systems” that enable teachers to deliver personalized learning. 
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They call for a shift in pedagogy to much of learning time spent online with 
teachers focused less on providing knowledge and more on assisting students to 
apply knowledge. 

4. (2016) Graeme Atherton, Constantino Dumangane, & Geoff Whitty: 
Charting equity in higher education: Drawing the global access map 

Focus: Draw attention to issues of equitable access to HE, create a global access 
map, and argue for improved data collection from those who participate in HE. 

Nature of Analysis: Report informed by secondary data analysis, a survey of 
HE experts in 50 countries and case studies in 6 countries. 

Key Findings or Arguments: Recommend a “Global equity data charter” to 
improve data collection on HE access and participation. This data should be 
collected and made open access. Governments, international organizations, and 
HE providers all have a role to play in understanding what actions need to be 
taken to make HE participation more equitable. 

5. (2016) EdSurge: Decoding adaptive 

Focus: Defines adaptive learning and discusses the potential benefits and 
challenges of using adaptive tools in the classroom. 

Nature of Analysis: Taxonomy informed by literature review, interviews 
with educators, product designers and academics, and reflection of adaptive tools. 

Key Findings or Arguments: Adaptive learning needs a clear educational 
vision, will include teachers and technology in partnership (where adaptive tools 
can be an ever present teaching assistant), and provide educator friendly data that 
can be collected and collaborated across all adaptive tools and against a common 
set of data standards. 

6. (2016) Rose Luckin, Wayne Holmes, Mark Griffiths, & Laurie Forcier: 
Intelligence unleashed: An argument for AI in education 

Focus: Outline AI in education (AIEd) and what it can offer learning now and 
into the future. 

Nature of Analysis: Opinion piece informed by literature detailing the 
importance of AIEd in education. 

Key Findings or Arguments: AIEd will play a critical role in education 
system reform, needs to function in blended learning spaces (digital and “tradi- 
tional” classroom activities), and involves teachers and learners in the co-design of 
appropriate AIEd. Calls for focusing AIEd funding, developing infrastructures, 
and creating demand for AIEd technologies. 

7. (2017) Barbara Means, Robert Murphy, & Linda Shear: Building efficacy in 
learning technologies, Volume 1 

Focus: Understanding the efficacy of learning technologies to enhance learning. 

Nature of Analysis: Literature review that makes recommendations about 
identifying, planning, implementing, and evaluating learning technologies. 

Key Findings or Arguments: Learning technologies are just one part of an 
instructional system, not as learning intervention in itself. This report makes 
recommendations about how the effectiveness of learning technologies can be 
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evaluated and explores the importance of technology implementation by teachers 
for a successful impact on learner outcomes. 

8. (2018) Joe Deegan & Nathan Martin: Demand-driven education: Merging work 
and learning to develop the human skills that matter 

Focus: Demand driven education needs to take account of the global econ- 
omy and be technology-infused, gig-oriented, and industry-driven. 

Nature of Analysis: Report informed by interviews with 20 education and 
workforce experts in the US and UK. 

Key Findings or Arguments: Education systems need to develop specific 
skills for complex thinking; utilize dynamic digital pedagogies while preparing 
educators to embrace new forms of teaching and learning; respond to needs of 
the labor market; create flexible and adaptive pathways for learning; and support 
changes that make the education landscape function better so that it enables tra- 
ditional and alternative providers to participate. 

9. (2018) Jeffrey J. Selingo: The future learners: An innovative approach to under- 
standing the higher education market and building a student-centered university 

Focus: Consumer model of HE requires “segmentation” to build new aca- 
demic offerings and personalize campus services. 

Nature of Analysis: Report informed by a survey of 2,600 participants aged 14—40. 

Key Findings or Arguments: Institutions need to design more flexible 
pathways and offer alternative credential, certificates, and a broader range of 
delivery methods (students prefer independent learning, and technology greatly 
enhances their learning experiences). Need to move away from a “one-size-fits- 
all” system and better understand the diversity of students’ needs. 

10. (2018) Jeffrey J. Selingo: The networked university: Building alliances for inno- 
vation in higher education 

Focus: How alliances between universities can develop strategic solutions — 
using technology — to solve HE’s toughest problems related to access, retention, 
competition, and making good on the promise of digital education tools. 

Nature of Analysis: Report informed by interviews — no mention of how 
many interviews were conducted or who with. 

Key Findings or Arguments: The networked university will allow students to 
access courses at their institution and others across the consortium, and courses will be 
blended with a mixture of online and face-to-face content. Communication occurs on 
a shared network, underpinned by a digital library and personalized, on-demand vir- 
tual tutors. Essentially, universities work together to build a digital infrastructure that 
supports student opportunity to learn more flexibly in a largely online environment. 


Digital Transformation and the Disruption of Teaching, 
Curriculum, and Assessment 


As shown in the list of reports above, Pearson has commissioned policymakers 
and academics to produce “Open Ideas” reports that legitimize its corporate 
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vision for digital transformation premised upon disruptive changes to the teaching 
profession, the delivery of curriculum, and assessment. In other words, Pearson 
is seeking to disrupt the three message systems of schooling (Bernstein, 1977) 
and, more broadly, the organization and function of education as a public good. 
While many have called for reform of schooling to modernize this 19th-century 
institution, particularly in regards to the provision of homogeneous curriculum, 
age-based learning, and traditional models of teacher-led instruction, Pearson is 
betting strongly that such reforms, coupled with its focus on digital approaches, 
including the development of AIEd, will pay off. In this section, we consider 
how Pearson is actively working to disrupt existing educational practices in 
order to make markets for its products and service and, in turn, the role of the 
“Open Ideas” reports in this strategy. We focus specifically on Hill and Barber’s 
(2014) “Preparing for a renaissance in assessment” report, which is illustrative of 
the key messages that we have identified across the 10 reports in our review. 


Disrupting Teaching 


Pearson publicly promotes its support for teachers in its marketing, and through 
a range of programs and awards. However, its corporate vision, and the impli- 
cations it may have for teachers, presents a more ambivalent picture. For 
example, Hill and Barber (2014) have argued that teaching needs to be 
transformed 


from a largely under-qualified and trained, heavily unionized, bureau- 
cratically controlled “semi-profession” into a true profession with a dis- 
tinctive knowledge base, a framework for teaching, well defined common 
terms for describing and analyzing teaching at a level of specificity and strict 
control, by the profession itself, on entry into the profession. 


(p. 20) 


Hill and Barber state that teaching is currently an “imprecise and idiosyncratic 
process that is too dependent on the personal intuition and competence of indi- 
vidual teachers” (p. 38), implying that teachers cannot be trusted. They suggest 
that this challenge can be addressed by “overthrowing” and “repudiating” the 
“classroom teacher as the imparter of knowledge” and replacing them with 
“increasing reliance on sophisticated tutor/online instruction” (p. 23). This is just 
one example of Pearson commissioning an analysis that paves the way for the 
personalized learning approaches that the company is developing. While Pearson 
does not call for replacing teachers, it has aligned the company with the view that 
teaching will be transformed by AIEd and this will require new kinds of profes- 
sional expertise (Luckin et al., 2016). For example, Hill and Barber (2014) envi- 
sion teachers as professionals who complement personalized learning software by 
providing guidance, coaching, motivation, and management of students. 
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However, Pearson also endorses the deprofessionalization of teaching that has 
become popular in low-fee private schools in Sub-Saharan Africa, India, and parts 
of South-East Asia. Pearson is reducing the need for trained teachers, and conse- 
quently, the cost of teacher salaries for schools and school systems. Paying 
appropriate teacher salaries is a major obstacle for the profitability of low-fee 
private schools in the Global South. Pearson looks set to continue its support, 
explicitly or implicitly, for low-fee schools provided by companies like Bridge 
International Academies, in which staff are required to read prefabricated lessons 
word-for-word from a tablet device. Staff in these schools must not deviate from 
the script and must implement learning activities in a step-by-step fashion 
(Renshaw, 2017; Riep, 2017b). The routinization of teachers work increases its 
susceptibility to automation, rather than promoting complementarity and new 
kinds of professionalism, clearly departing from the aims expressed in the EdTech 
focused aspects of Pearson’s marketing and corporate strategy. While arguing that 
the company is supporting the provision of education in contexts where it would 
not be available otherwise, Pearson is forced to produce highly contradictory 
messages about the different kinds of teacher professionalism it seeks to promote. 


Disrupting Curriculum and Instruction 


Replacing teachers as the central agents of learning will supposedly enable “truly 
personalized instruction” to be delivered (Hill & Barber, 2014, p. 21), which the 
researchers define as: 


instruction that is adjusted on a daily basis to the readiness of each student 
and that adapts to each student’s specific learning needs, interests and aspira- 
tions. The fundamental premises of personalized learning have been a part of 
the writings of educators for decades but have, in recent years, become a 
realizable dream, thanks to the advent of new digital technologies. 


(p. 56) 


Pearson’s vision for “next-generation learning” is based on the digital manage- 
ment of curriculum, learning resources, assessment, data, and analysis. Pearson 
Realize — a single sign-on platform for accessing resources, assessment, student 
data, and management tools — will make decisions about what students need to 
learn through the continual monitoring and assessment of data generated by their 
engagement with learning and assessment tasks. Learning resources will be pro- 
vided based on searches for materials “that most closely match students’ learning 
needs, accessing both purpose-built, commercially available materials and the 
rapidly expanding collections of public-domain and creative-commons resources” 
(Hill & Barber, 2014, p. 54). 

These next generation learning systems, Pearson argues, will “create an explo- 
sion in data” from the continuous tracking of individual students (Hill & Barber, 
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2014, p. 55). Information generated through these learning systems, through the 
application of data mining and data analytics, will be used to “revolutionize” 
educational research and generate evidenced-based strategies for teaching and 
learning (Hill & Barber, 2014). Such partnerships potentially increase the market 
share for both companies, as well as the number of users integrated into both 
platforms, which in turn increases the amount of data that can be generated and 
joined up. 

There are a number of issues to consider in relation to Pearson’s collection and 
use of digital data, including questions about privacy, consent, ownership, trans- 
parency, bias, as well as openness. Pearson collects a range of data from customers, 
including names, phone numbers, addresses, birth dates, jobs, course information, 
personal interests, credit card and billing information, shopping selections, and 
data about activity on their websites. Pearson also collects a range of educational 
data, including assignments, student coursework, responses to interactive exer- 
cises, scores, grades and instructor comments, and details of the books the custo- 
mer has read or activities the customer has completed. These data are de- 
identified and aggregated to audit and analyze how Pearson’s services are used, to 
conduct educational research, and to support strategic development of its pro- 
ducts and services. Pearson may also share data with institutions that purchase its 
services, with other companies in its group, and with companies that purchase its 
business assets. As a joint report by the British Academy and The Royal Society 
(2017) notes, “the ability to protect personally identifiable information is an 
essential component of trustworthy organisations. However, this can be difficult, 
if not impossible to achieve, even with the help of advanced privacy preservation 
techniques” (p. 31). The risk of data breaches has become part of the lifecycle of 
large education technology companies. 

Consent to collect and use the various kinds of data outlined above is not 
always explicitly sought. Pearson’s privacy notice specifies that consent will be 
sought to share or use data in ways not covered by the notice, or to send mar- 
keting material to customers under 16 years of age. While users actively give 
consent to sharing their data when they enter it into online forms, for example, 
Pearson also collects other data about user interactions with its websites and 
platforms, and users may be less aware of this information collection or the con- 
sent that is given through use of the service. Moreover, the extent and nature of 
data collection by Pearson makes it difficult for users of its services to understand 
exactly what data are collected and for what purposes. 

This relationship can complicate questions about data ownership and responsi- 
bility, as well as accountability for good management of data. Moreover, much of 
the data that Pearson generates from the use of its services, and thus has owner- 
ship of, does not appear to be openly available. If these data remain locked up in 
private corporate silos, then their potential benefit for all learners and society 
more broadly will not be realized. Pearson’s development of digital education 
services is effectively privatizing aspects of the emergent global data infrastructure 
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of education, the social benefits of which would be greatly enhanced if it were 
open and shared. 


Disrupting Schools 


The changes described above suggest that students will increasingly sit at 
computers for personalized instruction, reducing the need for brick-and- 
mortar institutions. In fact, Hill and Barber (2014) argue that schools reflect 
the demands of an outdated agrarian society, where the short hours of the 
school day and long periods of school holidays are not aligned with the needs 
of parents and guardians. Pearson’s Connections Academy, for example, 
already offers tuition-free virtual schools for K-12 students. The website for 
Connections Academy states that “in this virtual classroom, students can spend 
the school year reaching their highest potential through a uniquely indivi- 
dualized learning program” (Connections Academy, 2019, para. 2). While it is 
free for students to attend these public charter schools, in some Connections 
Academies there are costs for field trips — which offer important social 
opportunities for students and teachers to meet face-to-face — as well as school 
supplies, including computers and their maintenance. These schools are 
rapidly expanding in the United States, serving more than 70,000 students 
across 27 states in 2017/18. 

Pearson also offers a fully accredited education from the United States to any 
student in the world via its International Connections Academy. Pearson’s 2017 
annual report noted that the company is currently the second largest provider of 
virtual schools in the US, and there is a need to capitalize further on the fact that 
virtual schools are permitted in 34 states, covering 80% of the K-12 population, 
including the “big three”: California, Texas, and Florida. Currently, virtual 
schools make up only 6% of Pearson’s total sales (£274m), but it estimates that 
the current market is worth US$1.5 billion and states that it is growing rapidly. 

Pearson’s new business strategy aims to accelerate the shift towards reduced 
need for teachers and schools in order to grow the market for data-driven per- 
sonalized learning that is provided direct to consumers across their life course. 
Pearson’s vision for education is one in which the cost and contribution of 
teachers to public education is reduced, while the company plays a more central 
role in education provision globally through its new platforms and the data 
upon which they run. We argue that Pearson’s vision encourages the broad 
privatization of schooling. Its approach to individualizing learning potentially 
locks customers into proprietary online services (as opposed to owning and 
having use of textbooks in perpetuity) and follows similar models used by other 
large digital services providers, which are based on subscriptions and retaining 
control over content. Pearson is likely to provide off-the-shelf generic services 
that plug in to third-party systems, encouraging the growth of a private digital 
education ecosystem. 


120 Hogan and Sellar 


Conclusion 


During the 19th and 20th century, governments and private companies built the 
infrastructure of modernity, such as roads and rails, pipes and wires. Building data 
infrastructure has been arguably the major project of the 21st century (Mayer- 
Schönberger & Cukier, 2013), and opening this infrastructure to a range of sta- 
keholders creates opportunities for important advances in knowledge and the 
provision of public and private services. There is clearly a place in the GEI for 
private providers of education services, and private companies are generally better 
placed to provide a range of technical services that will underpin the next gen- 
eration of teaching and learning. Moreover, providing profitable services in the 
interests of its shareholders is a reasonable objective for any edu-business. 
However, Pearson’s vision for the company that it wants to become raises two major 
causes for concern in relation to the integrity and sustainability of public schooling: 


1. the privatization of data and data infrastructures, which enclose innovation 
and new knowledge about how we learn, turns public goods into private 
assets; and 

2. the transformation and potential reduction of the teaching profession 
diminishes the broader purposes and outcomes of public schooling in favor 
of personalized learning that focuses on individual knowledge and skills. 


Both of these issues arise from a particular approach to profiting from educa- 
tion, which Harvey (2004) described as accumulation by dispossession. Pearson is 
following the example of other technology companies by seeking to disrupt 
public schooling in order to privatize educational data infrastructures and profit 
from the data that it accumulates through the provision of its services. These data 
will be the “lifeblood” of new education platforms and will be crucial for gaining 
new insights into how we learn. Pearson’s corporate strategy exemplifies “the 
corporatization and privatization of hitherto public assets (like universities)” and is 
part of “a new wave of ‘enclosing the commons’” (p. 75). 

Pearson’s vision for education laudably promotes the benefits of technological 
developments and their combination with new kinds of teacher professionalism 
and new insights produced by the learning sciences. However, Pearson’s efforts to 
contribute to the disruption of teaching and public schooling, including in the 
Global South, and its development of platforms that produce new volumes and 
varieties of education data, while raising new ethical concerns about openness, 
privacy, bias, and transparency, will create significant risks for public education. 
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COSTS OF BIG DATA 


Challenges and Possibilities of Cost-benefit 
Analysis of ILSAs 


Laura C. Engel and David Rutkowski 


Introduction 


As part of larger global trends in the rise of big data and the datafication in education 
systems, the proliferation and prominence of international large-scale assessments 
(ILSAs) appear to rest on a basic idea that more (cross-national) evidence will lead to 
optimal solutions of (national) educational problems. While this rather basic idea that 
empirical knowledge and the ability to benchmark internationally yields better out- 
comes is not new, the rather narrow application of achievement outcomes to opti- 
mizing the quality and effectiveness of education system performance has become a 
defining characteristic of the so-called global turn in education. Since the 1990s, 
more systems around the world, including low- and middle-income countries, are 
participating in international, regional, and national learning assessments (Kamens & 
Benavot, 2011). Results from these assessments and especially from flagship ILSAs, 
such as the Programme for International Student Assessment (PISA) and the Trends 
in International Mathematics and Science Study (TIMSS) have emerged as important 
indicators of achievement and educational success in systems around the world. 
These ILSAs readily inform policy decisions undertaken, fueling new data infra- 
structures and platforms for policy purposes, ultimately leading to “new govern- 
mental constellations” (Hartong, 2016, p. 524). In interesting ways, therefore, ILSAs 
are both driving factors and resultant effects of larger trends in the datafication of 
educational governance. 

ILSAs are generally a highly aggregated snapshot of students’ ability levels in a 
given age or grade. Given the formidable scope of international assessment pro- 
grams, such as the Progress in International Reading Literacy Study (PIRLS), 
PISA, and TIMSS, creative assessment designs are employed to allow for broad 
coverage in several content domains and so that assessed groups are measured 
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with appropriate precision. To that end, ILSA programs commonly use a 
sophisticated assessment design whereby each individual student is administered 
just a small number of the total possible items; yet, all items are administered 
throughout each of the reporting groups. Importantly, this type of assessment 
design is viable given the desired level of inferences: population and subpopula- 
tion as opposed to inferences at the individual level. To contextualize achieve- 
ment, depending on the assessment, information from the students, parents, 
teachers, principals, and national coordinators is collected. These data serve as a 
rich source of information regarding the context and correlates of learning. 
Background questionnaires seek information on, for example, affective and 
behavioral measures of students, teacher beliefs and practices, the home learning 
environment, and principal perspectives on school safety and resources. 

Since the mid-1990s, ILSAs have continued to develop into what many in the 
policy community view as an essential tool that provides them with insights into 
how a system is doing with respect to others (Grek, 2009; Lockheed & Wagemaker, 
2013). This is especially true in the United States (US), where policymakers have 
frequently sought out results from ILSAs to gain new insights and signal a sense of 
where it stands vis-a-vis other systems, often regarded as competitors. 

While much has been written about ILSAs, in particular the uses, motivations, 
misuses, and limitations of these assessments, there has been only minimal 
accounting of the costs of participation (Engel & Rutkowski, 2018; Wagner et al., 
2011; Wolff, 2007). Specifically, existing research has focused on some of the tan- 
gible costs of ILSAs in different settings (Wolff, 2007), including the role of bilateral 
and multilateral donor organizations that provide funding assistance to systems to 
participate in ILSAs, and develop direct policy advice to systems based on ILSA 
findings (Kijima, 2010). More recent research focused on direct and indirect costs 
of the PISA in the US revealed several interesting internal components about how 
PISA is organized and governed (Engel & Rutkowski, 2018). In particular, findings 
from this research suggested that costs include the overhead that systems pay to the 
Organization for Economic Co-operation and Development (OECD), and the 
costs of national implementation of ILSAs, which include drawing a sample, 
recruiting schools, administering the assessment, scoring open-ended items, and in 
the US, incentivizing participation of schools, teachers, and students. 

A logical question emerging from this existing research is whether the costs 
associated with ILSA participation are worth the touted benefits. This seemingly 
obvious question, however, is fraught with complexities that make answering it a 
difficult, if not an impossible, endeavor. Building on previous research, in this 
chapter, it is our aim to take up and explore some of these challenges and 
opportunities associated with research on the monetary costs of participating in 
ILSAs. To do so, we focus on US participation in ILSAs, providing a brief over- 
view of the history of US participation in ILSAs and what we know from the 
data related to US costs of both the PISA and TIMSS assessments. We con- 
centrate on the example of the US as these costs are publicly available, but also 
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because the country has been a driving force behind the initiation, development, 
and circulation of ILSAs globally. We then develop a discussion around core 
issues facing a cost-benefit analysis (CBA) of ILSAs, arguing that it is highly likely 
to be a futile endeavor due to the challenges in quantifying benefits and costs in 
the ways that CBAs require, and because CBAs tend to oversimplify situations, 
creating a narrow, economistic, and nationalistic view of what a system gains 
from participating in an ILSA. While we argue that CBA related to ILSAs may be 
problematic, we nonetheless frame costs of ILSA as a meaningful focus of future 
research. Specifically, by examining costs of ILSAs, we argue that critical ques- 
tions are revealed related to how the architecture of ILSAs are constructed and 
become mobilized in different settings. 


US Participation in ILSAs: What Does It Cost? 


The US has participated in ILSA programs since they were initially developed, starting 
from the early International Association for the Evaluation of Educational Achieve- 
ment (IEA) studies in the late 1950s. The primary purposes of ILSAs within the US 
context are concisely described by the National Center for Educational Statistics 
(NCES) in the US Department of Education call for proposals for the International 
Early Learning Study Pilot: ILSAs “provide comparable indicators on student perfor- 
mance and schooling practices across countries in order to benchmark US student 
performance” (NCES, 2017, p. 11). Plisko (2013), the former Associate Commissioner 
at the National Center for Educational Statistics, described the US engagement in 
ILSAs as derived from “both a legal mandate and a history of international collabora- 
tion among governments and among researchers” (p. 326). She notes several factors in 
vetting proposals for participation in ILSAs, including other budgetary priorities, other 
surveys, technical issues in administering the survey, and issues related to comparability 
and generalizability. To a certain extent, participation in ILSAs is also driven by the 
other systems that are participating. Plisko (2013) identifies that US participation 
attracts the participation of other nations. Additionally, 


. the US looks for signals from the major European countries and from 
other English speaking countries as to their willingness to engage in the 
assessments. Something of a critical mass is necessary to ensure sufficient 
resources to help defray the international costs and enable meaningful com- 
parisons of results. 

(p. 327) 


Although the US has participated in a full range of ILSAs, there are a few 
instances where they declined to participate. For example, the US has not parti- 
cipated in the last several rounds of the International Civic and Citizenship Study. 
Compared with IEA studies, the OECD studies work differently, as OECD 
member states are obligated to participate in PISA. 
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In both TIMSS and PISA, a number of US states have also participated as inde- 
pendent “adjudicated systems” where they also receive state-level results (Engel & 
Frizzell, 2015). There has been little consistency in state participation in studies over 
time. In PISA, these have included Connecticut, Florida, Massachusetts, and North 
Carolina in previous cycles. Other states have participated in previous cycles of 
TIMSS, including the nine (Alabama, California, Colorado, Connecticut, Florida, 
Indiana, Massachusetts, Minnesota, and North Carolina) who participated in TIMSS 
2011 as part of the NAEP-TIMSS Linking Study. 

For this chapter, we will focus on the two largest ILSAs that the US partici- 
pates in: TIMSS, which assesses mathematics and science knowledge and skills 
acquired by fourth and eighth graders, and PISA, which measures mathematic, 
scientific, and reading literacy of school-enrolled students who are 15 years of 
age. Table 7.1 provides a general overview of the studies, who and what are 
tested, how often, the sponsoring organizations, and how many systems partici- 
pated in the last cycle. 

Important for this chapter is that TIMSS is administered in two cohorts (grades 4 
and 8) and PISA is administered in one cohort (age 15). Also, PISA assesses three 
subjects with one of those subjects considered the major domain and the other two 
are the minor domains. The major domain alternates each cycle and includes more 
questions on the assessment. Some of the questions on the background ques- 
tionnaire focus on that topic. One disadvantage of such a design is that it is not 
advisable to evaluate trend results from major domain to minor domain, meaning 
that countries can only validly assess their trend in any one subject over a nine-year 
period (Rutkowski & Rutkowski, 2016). TIMSS, on the other hand, equally 
assesses science and mathematics, allowing for trend analysis on both domains every 
four years. Also, TIMSS includes background questionnaires that focus on both 
subjects. Interested readers who wish to know more about the studies are encour- 
aged to visit the studies’ websites, which provide a great amount of detail on the 
studies and what is collected. For this chapter, readers should be aware that the 


TABLE 7.1 General overview of ILSAs 


PISA TIMSS 

How often are tests conducted? Every 3 years* Every 4 years 

Who is tested? 15-year-old students 4th and 8th grade 

students 

What is tested? Ability to apply skills and Mathematics and 
competencies to “real-world” science curriculum 
contexts 

Who sponsors the test? OECD IEA 

How many systems partici- 79 (2018) 64 (2019) 


pated in the last cycle? 


Note. *Due to the COVID-19 global pandemic, PISA has been delayed one year until 2022. 
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studies are significantly different, and that while we focus on costs of both TIMSS 
and PISA, a direct comparison between these two assessments is not advisable. In 
the following section, we detail our research concerning the costs of ILSA partici- 
pation for the US at the federal level. 


ILSA Costs in the US 


Information on costs of ILSA in the US was collected from publicly available docu- 
ments (including open data on US participation costs) from the US Department of 
Education’s National Center for Education Statistics (NCES) reports to the US 
Office of Management and Budget. NCES, by mandate of the US Congress, is the 
major federal agency responsible for the collection and analysis of education data in 
the US and internationally; “NCES fulfills a Congressional mandate to collect, col- 
late, analyze, and report complete statistics on the condition of American education; 
conduct and publish reports; and review and report on education activities inter- 
nationally” (NCES, n.d., para. 1; see also Plisko, 2013). Every time the US conducts 
a study, a package of materials is posted to the Federal Register for public comment, 
including the plan for the study, what the study includes, the design and rationale, 
and how much it costs the government. These proposals are “vetted through the 
annual budget process, as planning for participation must begin some years in 
advance of administering the assessment” (Plisko, 2013, p. 327). 

In this chapter, we focus on PISA 2012, PISA 2015, and TIMSS 2015. We also 
collected national and sub-national policy statements about PISA participation, 
recruitment letters, and international technical reports and websites. As a secondary 
source used to triangulate information collected via public online sources, we also 
drew on data from interviews carried out with four government officials at the state 
and federal level in the US: two officials in the US Department of Education and 
two officials from state-level Departments of Education from PISA 2012 state-level 
participants (Connecticut and Florida: Engel & Rutkowski, 2018). 

In looking at this information, the cyclical nature of PISA and TIMSS inter- 
national assessments means that their true cost per a single administration of the 
assessment is not easily defined. Money spent in one year may be applied to a 
previous assessment and to a future assessment; therefore, figures provided by the 
NCES span multiple years (see Table 7.2). 

Each assessment cycle also includes in-kind burdens. These costs include admin- 
istrators’ time, teachers’ time, and students’ time for preparing, administering, and 
taking each test. While these in-kind burdens are small compared to the total cost of 
the assessments, they are included in the total cost of the assessment cycle. In the 
supplemental documents provided by NCES for the PISA 2015 Main Study, PISA 
2015 Field Test, TIMSS 2015 Main Study, and TIMSS 2015 Field Test, a cost for 
administrators, teachers, and students is calculated based on the number of hours each 
individual would be expected to participate. To account for time spent preparing for, 
administering, and taking PISA tests, NCES allocated $50.00 for each administrative 
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TABLE 7.2 Reported overall costs for PISA 2015 and TIMSS 2015 


Study Cycle cost Number of Annual cost 
years 

PISA 2015 Main Study $ 4,278,915 4 $ 1,069,728 
PISA 2015 Field Test $ 2,483,434 2 $ 1,241,717 
TIMSS 2015 4th and 8th Grade $ 10,658,124 5 $ 2,131,624 
Main Study 

TIMSS 2015 4th and 8th Grade $ 2,578,570 1 $ 2,578,579 
Field Test 


All figures are from supplemental documents submitted by the NCES and publicly provided on the 
Federal Registrar. 


hour, $35.00 for each school coordinator hour and each teacher hour, and $7.25 for 
students. The cycle cost in Table 7.2 reflects the in-kind burdens for administrators, 
school coordinators, teachers, and students (only for the school questionnaire). The 
majority of student time — spent on test directions and taking the assessment — is not 
factored into in-kind burdens for students. An independent calculation of student 
time for directions and assessment-taking time slightly increased the in-kind costs 
associated with PISA and TIMSS (see Table 7.3). 


Possibilities and Challenges of Exploring Costs and Benefits 
of ILSAs 


As we uncover new information about the cost of ILSAs like TIMSS and PISA, 
the logical turn is whether they are, in fact, worth the investment. We consider 
the cost both in terms of how actual taxpayer dollars are spent, but also the 


TABLE 7.3 Calculated overall costs for PISA 2015 and TIMSS 2015 


Study Reported cycle Additional cost for Number Annual cost 

cost calculated student of years 

hours 

PISA 2015 Main $ 4,278,915 190,798.25 4 $ 1,117,428 
Study 
PISA 2015 Field $ 2,483,434 45,638.75 2 $ 1,264,536 
Test 
TIMSS 2015 4th $ 10,658,124 262,783.50 5 $ 2,184,182 
and 8th Grade Main 
Study 
TIMSS 2015 4th $ 2,578,570 47,545.50 1 $ 2,626,125 
and 8th Grade Field 
Test 


Note. These costs are not included because those hours are not subject to the Paperwork Reduction 
Act of 1995. 
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burden of these assessments on students, schools, and state systems. On the one 
hand, the costs detailed in the previous section are relatively small in relation to 
the expansive US federal educational budget of approximately US$60 billion per 
year (US Department of Education, 2019). On the other hand, we acknowledge 
that in a system of widening inequalities, the resources allotted to ILSA partici- 
pation is in fact a significant amount, and that a judgement of the merit and 
worth of ILSAs is prudent. 

In the US system, as with many other OECD countries, one popular method 
of judging the merit and worth of systems such as ILSA is by employing a cost- 
benefit analysis (CBA) approach. CBA is the practice of quantifying the total 
monetary costs and the total monetary benefits of a policy, program, or project. 
Largely based in both empiricism and economic thought, the underlying goal is 
to find a way to maximize social welfare through quantification and comparison. 
Normally, costs and benefits are determined and then converted into monetary 
units (e.g., dollars, euros, yen). By quantifying both units onto the same scale, 
analysts are empowered with mathematical properties where, for example, they 
can subtract the costs from the benefits to determine the overall worth of the 
program. Further, the conversion to monetary units also allows for cross program 
comparison. So analysts could study the costs and benefits of national participa- 
tion in an ILSA, funding a universal school lunch program, and providing com- 
puters to all students. In each scenario, results can be compared side by side and 
the most efficient project can be funded; or so the theory goes. 

Calculating all costs and benefits is not for the light hearted, and there are 
many factors that go into such a process. In fact, the field has developed to 
include a range of models and processes each resulting in their own monetary cost 
and benefit. Each approach to CBA incorporates a different way of addressing 
critical questions that emerge such as: Can all costs and benefits be determined? 
Should both direct and indirect costs be counted? Are benefits a direct or indirect 
result of the program? Should indirect benefits be discounted and if so, by how 
much? Interested readers are referred to Boardman et al. (2017) and Gramlich 
(1990) for comprehensive treatments of methods and techniques for developing a 
CBA. 

Shapiro and Schroeder (2008) explain that CBA represents an empirically based 
rational decision method in the policy sciences. These authors further claim that 
CBA is rooted in the positivist notion of science, which was adopted by many in 
the policy analysis community to help legitimize the field. In particular, within 
positivism scientific knowledge only results from observation and empirical ver- 
ification. Further, for rational and unbiased policymaking to exist, it is necessary 
for policymakers to have unbiased and rational policy tools to make decisions (a 
hallmark of positivism). CBA is thus designed to meet the rigor of the scientific 
process in a social setting. As such, for CBA to properly work, all benefits must be 
defined and quantified. If, for example, a benefit is not entirely quantifiable, it 
cannot enter the model and does not exist in the eyes of the analyst. 
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Levin and McEwan (2001) recognize the limits of analyzing all costs and ben- 
efits in monetary terms: 


Only under certain circumstances would one wish to use cost-benefit ana- 
lysis. Those situations would obtain when the preponderance of benefits 
could be readily converted into pecuniary values or when those that cannot 
be converted tend to be unimportant or can be shown to be similar among 
the alternatives that are being considered. 


(p. 15) 


An additional limitation of CBA is that defining benefits is easily biased by an 
analyst’s policy preferences or life experiences. In fact, the idea of placing a 
socially constructed value to a benefit is neither objective nor unbiased, and open 
to many different interpretations. Of course, in the US, with a population of over 
325 million people, simply defining the groups that benefit (let alone are harmed) 
by the assessments could be overwhelming. For instance, would one include all 
policymakers at the federal and state level? In the US, where local school boards 
have a large influence over educational decision-making, should they also be 
included in the calculation? Moreover, the results from ILSAs are used outside of 
the policy community and are a frequent topic in educational research. As such, 
would it be appropriate to include the benefits to the research community? 
Without adequately being able to address these complexities, CBAs risk over- 
simplification, where simple calculations of benefits end up creating narrow, 
economistic, and nationalistic views of what a system gains from participating in 
an ILSA. 

Although we have raised concerns over CBA, we fully believe that there are 
important advantages and opportunities to focus on the costs of ILSA participa- 
tion. In fact, we favor a larger and more extensive evaluation of ILSAs, which 
closely examines whether these assessments meet, or as may be the case possibly 
undermine, the needs of the educational community. By explicitly engaging in an 
evaluation of ILSAs at the system level, it is possible to pave a way forward in the 
hope of helping national governments determine whether ILSAs are meeting 
system-level goals and whether individual systems are well-served by continued 
participation. One issue with evaluating ILSAs, however, is that the goals of 
participation are often poorly stated and difficult to evaluate. For example, the 
NCES provides two purposes for participating in ILSAs: “To learn about the 
performance of US students and adults in comparison to their peers in other 
countries; [and] To learn about the educational and work experiences of students 
and adults in other countries” (NCES, 2019). Such broad goals make it nearly 
impossible to truly judge if the assessments are meeting the needs of the system. 

As such, we would encourage the US to clearly describe the purpose for par- 
ticipating in each study, allowing for clear analysis of the stated goals. To that 
end, we would also invite a more robust dialogue and debate from the education 
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community about decisions not to participate, such as in the case of the International 
Civic and Citizenship Study. Second, we also recommend that the US government 
consider the needs of the research community that often spends extensive amounts 
of time and energy engaging with the secondary data emerging from ILSA to inform 
both research and policy. More active participation from stakeholders (policy and 
research) in developing what should be collected and why may be key to improving 
ILSAs and evaluating their larger worth to society. 

Additionally, while we see CBA as limited with respect to ILSAs, we would 
argue that through a focus on the actual costs of ILSAs, it is possible to generate 
new knowledge about the ways in which ILSAs contribute to mobilizing new 
governance structures and generate varied uses and impacts across education 
policy spaces. This seems particularly valuable as a greater number of economies, 
including those that do not have national assessments and for whom ILSAs may 
become cost-effective assessment tools at the country level, are participating. We 
would invite additional cross-national investigations that generate new knowl- 
edge about costs of ILSAs, which are fruitful in better deciphering how the 
architecture of ILSAs are both constructed and mobilized in different settings. 
Key to understanding trends in datafication of education systems, examining costs 
of ILSAs allow us to critically investigate elements such as: (1) what policy actors 
identify as key driving factors of participation; (2) how systems mobilize resources 
to participate; and (3) what organizational governance structures surround ILSA 
participation. In each of these dimensions, costs help illuminate factors that bolster 
critical understandings of ILSA. 

We would also like to propose that transparency of ILSA costs helps illuminate 
the linkages and social relationships between various actors, from international 
organizations like the OECD to government officials, edu-businesses, policy 
entrepreneurs, and other non-governmental actors (Ball, 2012; Lingard & Sellar, 
2013). This is especially timely given that the scholarship on the rise and expan- 
sion of ILSA has begun to map the formation of networks and interrelationships 
across multiple educational stakeholders and places where education policy is 
influenced and made (Sellar & Lingard, 2014). Moreover, understanding costs 
may illuminate how policy flows across multiple sites and builds new knowledge 
about “the production, multidirectional flows and usages of data in forms of 
comparison and governance in education within new spaces produced by globa- 
lization” (Gulson et al., 2017, p. 229). By examining the costs of TIMSS and 
PISA within the US context, a couple of interesting features are revealed in how 
major ILSA are governed and utilized in the US context. For example, these data 
reveal the utilization of contractors and sub-contractors, such as Westat, Pearson, 
and Windwalker Corporation. Examining costs reveals also that US schools, 
school coordinators, selected teachers, and students are each offered financial 
incentives for their participation. 

In light of the above considerations, we have three recommendations. First, 
countries need to be more transparent concerning the costs of participating in 
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ILSAs. The research presented in this chapter was time consuming to collect and 
only uncovered after multiple interviews with government officials and deep 
dives into complex budgets. Such complexities will be exponential in participat- 
ing systems with funding structures that include international/regional organiza- 
tions and/or private donors (Kijima, 2010; Rutkowski & Engel, 2010). 

Second, while it may be the case that ILSAs pose minimal monetary costs for most 
participants, they do require resources, and it may be the case that those resources 
could be better allocated. As such, taxpayers, students, teachers, and administrators 
participating in assessments have a right to know the exact reasons that governments 
are expending resources to participate in ILSAs and what is to be gained from parti- 
cipation. Ambiguous goals and purposes for participation result in less accountability 
allowing governments to pick and choose when and how they use the assessment 
results. We are not suggesting that we monetize the benefits, but rather propose the 
need for clear and measurable goals. One example of such a process can be found in a 
framework proposed by Oliveri et al. (2018). This framework is designed to help 
countries consider why they are participating in ILSAs and if the given ILSA can 
meet the participants’ needs. The process ensures that countries enter ILSAs with 
a clear purpose that can later be evaluated to ensure that the ILSA has met the 
needs of the country. Greater transparency also becomes important so that sys- 
tems understand the penalties of not participating or only partially participating in 
ILSAs (Liu & Steiner-Khamsi, 2019). 

Lastly, existing research has made evident that there is misuse of ILSAs, and this 
misuse needs to be monitored and mitigated (Zhao, 2020). While we have some 
understandings of what and how policy actors use ILSA data, the field continues to 
need a critical group of policy scholars aimed at generating new knowledge about 
the dynamics of ILSAs as they relate to larger trends of datafication and the rise of big 
data, and to ensure that governments are using the data correctly (Singer et al., 2018). 
Again, we are not suggesting that such information can or should be quantified; 
however, the misuse of ILSA data is an important thing to consider when judging 
the merit of ILSAs. 


Conclusion 


Our aim in this chapter was to focus on the US participation in two of the major 
ILSAs (TIMSS and PISA), detailing what we have learned to date about the costs. In 
discussing costs of system participation in ILSAs, we recognize the temptation to 
embark on a CBA. Undoubtedly, CBA has remained popular, largely because it serves 
the interests of a number of powerful constituencies, including the US government 
and the OECD, which are both proponents of the method. Our argument for this 
chapter is that when understanding ILSAs, calculating costs and understanding benefits 
may be noble endeavors, but the ability to truly calculate all costs and all benefits, as 
demanded by CBA, is futile and fraught with complexity and personal bias. We do, 
however, maintain that costs of ILSAs are a particularly fruitful set of evidence from 
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which to better understand ILSA from more of a critical governance perspective. In 
particular, we noted how costs could provide vital insight into how ILSAs are built and 
sustained in different contexts, including by whom, for what purposes, with what 
organizational and financial supports, and with what consequences. 
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DATA INFRASTRUCTURES AND THE 
(AMBIVALENT) EFFECTS OF RISING 
DATA INTEROPERABILITY 


Insights from Germany 
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Introduction 


With digital data playing an increasingly substantial role in educational practices 
and system monitoring, so too do the efficient and effective organization of these 
data. This organization refers to the networks of objects (the data itself, hardware, 
and software, but also policy “fragments,” such as educational standards or fund- 
ing formulas) and subjects (e.g., technicians, administrators, school actors, and 
intermediary agents) assembled around data and their socio-technical decontex- 
tualization, and recontextualization processes (data infrastructures; Hartong, 2018, 
p. 135). The implementation of such data infrastructures usually includes the 
advancement of data interoperability, which has been defined as the “ability of 
different information systems, devices or applications to connect, in a coordinated 
manner, within and across organizational boundaries to access, exchange and 
cooperatively use data amongst stakeholders” (Healthcare Information and Man- 
agement Systems Society, Inc., 2020, para. 1). Data interoperability provides, as 
reform promoters argue, tremendous benefits, including fast data exchange (i.e., 
for teachers who need information about mobile students, for districts or schools 
who need to fulfill ongoing reporting requirements), an easy way to adopt per- 
sonalized digital learning technologies to foster data-intensive research, and much 
more (US Department of Education, 2020). Pangrazio (2019) asserts that effi- 
ciencies can be achieved, saving time and money, if school data systems are made 


‘ 


interoperable so that they can “speak to” each other without a “human inter- 


mediary.” She continues that this has the advantage of enabling 


greater fluency of data across the various domains of a student’s schooling 
experience — from enrolment and behavioral information to learning and 
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reporting. Interoperability has the potential to streamline the sharing of 
information in ways that impact upon all aspects of student success and 
wellbeing. 


(para. 7) 


Driven by such potential, numerous initiatives and massive investments have emerged 
worldwide to increase data interoperability, including the implementation and/or 
expansion of so-called interoperability frameworks and/or standards. Examples include 
the globally active Schools Interoperability Framework (SIF) by Microsoft (now 
Access4Learning; see Sellar, 2017, SIF Association, n.d., Wyatt-Smith et al., 2019), the 
US-provider “Ed-Fi Alliance” (see Ed-Fi Alliance, LLC, 2020), but also a range of 
smaller or sector-specific initiatives and standard-promoting actors, such as the German 
Alliance for Education (see Biindnis fiir Bildung [bBfB], 2019). At the same time, data 
interoperability is also dependent on underlying technical infrastructure that allows data 
to be collected, processed, and used, which, as reform promoters bemoan, has 
remained widely deficient in many countries. 

In fact, such a functional yet often technically framed understanding of data 
infrastructures still shapes many debates on the opportunities (i.e., infrastructures 
are built and interoperability increased) and challenges (i.e., both are hindered) of 
digitalization (Forschler, 2018; Hartong & Förschler, 2019). Simultaneously, 
however, there is an emerging body of scholarly work that uses data infra- 
structures as a fruitful conceptual and methodological tool to trace the ongoing 
datafication and digitalization of education from a critical perspective (Sellar, 
2015, 2017). Equally, the focus is placed on the organizational process of datafi- 
cation (including data interoperability), yet this process is understood as carrying 
substantial power nested in the selective “relation-making” and the “in-forma- 
tion” of objects and subjects into “flowable” data (Hartong, 2020). In other 
words, attention is paid to the ongoing socio-technical (de)recontextualization 
practices (“infrastructuring”) needed to make data relational with other data. This 
is to legitimize these data as neutral, valid, and “holistic” representations of edu- 
cation, and to make these data relevant for educational actors. 

This relational understanding of data infrastructures also explains why they 
could be substantially disruptive for education, through the creation of a different 
form of policy or practical context “that ‘deforms’ the existing jurisdictional sys- 
tems and relationships” (Gulson & Sellar, 2018, p. 10). This can occur between 
educational actors, institutions, or systems, as well as between public and private 
or between state and non-state entities. Put differently, there appears to be a 
pressing need to understand the who and how of data “in-formation” and rela- 
tion-making, which includes multiple “data mediators” (Hartong, 2016) or 
“boundary brokers” (Williamson, 2016) acting within or around data infra- 
structures. This understanding also needs to consider the often-hidden logic of 
infrastructural rule-setting and valuation (Mau, 2019), which may deeply affect 
how education is governed based on infrastructural (de)contextualization. 
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Data interoperability standards are a good example for tracing such moments of 
rule-setting and valuation as well as (de)contextualization. This is because they 
build on the definition of meta-data and data models, which not only regulate 
which data can enter the infrastructure and be transferred from one place to 
another, but also how they can be algorithmically processed and, consequently, 
what type of information can ultimately be produced from the data infrastructure 
(Kubicek et al., 2019). It is standardized rule-setting (including defining data 
formats, data business rules or terminological frameworks of data collection) 
which makes data interoperability a moment of significant “disruption,” as it 
appears increasingly important with every investment in reducing data duplication 
and data alternatives for measuring “apparently” the same phenomenon. 

Additionally, while the initial development of data interoperability frameworks 
might involve substantial debate and multiple stakeholders, such frameworks have 
become “ready to use” products, often promoted by powerful alliances or aligned 
with already dominant commercial software products (such as Microsoft), and 
willingly accepted by educational actors who lack time and technical and financial 
resources. As Pangrazio (2019) observes, 


given the commercial tensions and competing interests in many areas of data 
and software use, the push for establishing interoperability in particular mar- 
kets and sectors often falls to industry-wide groups and community interests 
who act as catalysts for awareness-raising, campaigning and eventual broker- 
ing of agreed standards. 


(para. 5) 


While data infrastructures and data interoperability offer multiple fruitful entry 
points to increase critical awareness for their disruptive potential, this goes well 
beyond the rise of “big data,” but rather occurs for any kind of data collected 
through such infrastructures. Simultaneously, it seems important to not under- 
estimate the high level of context sensitivity associated with data infrastructures. 
For example, as more recent comparative data infrastructure analyses show, 
influences of national, sub-national, and local factors — such as specific account- 
ability regimes, politico-administrative traditions, or data protection legislation — 
can deeply affect the actual emergence, operation, and partly also deliberate 
restriction of data infrastructures (Hartong & Forschler, 2019; Lingard, 2019; 
Takayama & Lingard, 2019). A similar contextuality occurs when distinguishing 
between different types of data infrastructures, such as infrastructures for policy 
development, for accountability, or for school practice (Hartong et al., in press). 
There might also be great differences between different “stages” of development 
or sectors within individual data infrastructures, such as modelling, administration, 
piloting, data collection, or reporting (Hartong & Piattoeva, 2019; Kauko et al., 
2018). Finally, distinctions must be made regarding different types of data inter- 
operability, which not only include its level (for example, the interoperability of 
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data formats or of workflows), but also whether data collection is actually cen- 
tralized or if subsystems are made interoperable without data centralization. An 
example of this is the German Core Data Set (Kerndatensatz [KDS]), which is 
discussed in detail later in this chapter (see Kubicek et al., 2019). 

Following such a context-sensitive approach to better understand the disruptive 
potential of data infrastructures and interoperability, the goal of this chapter is to 
provide empirical insights into the German case which, when compared with 
other countries, can be regarded as a datafication and digitalization latecomer. 
Still, over the past decade, major transformations have been initiated, which have 
not only included significant expansion, standardization, and centralization of 
both administrative and performance data in school systems (Hartong & Forschler, 
2019; Lange et al., 2014), but also — and particularly since 2012 — a growing 
investment in digital teaching and learning tools. While we demonstrate how 
context-specific conditions of possibility (Savage, 2019) strongly mediated and 
partly restricted data infrastructuring in Germany, we also show that emerging 
data infrastructures have started to change these very same conditions and will 
most likely become more powerful in the future. We explicate this (see 
“Understanding the ‘disruptive’ potential of data infrastructures” section below) 
using the transformations of standardized testing, state school monitoring, as well 
as school platforms/learning management systems as examples. The chapter closes 
with a summarizing discussion and questions that are yet to be addressed. 


The German “Context” of Datafying and Digitalizing Education 


Different from countries that have already created extensive educational data infra- 
structures and systems of e-governance such as Australia, the United States, or the 
United Kingdom (Hogan et al., 2016; Koyama, 2011, Ruppert, 2012; Williamson, 
2016), in Germany, datafication and digitalization policies emerged much more 
cautiously. There are different context-specific reasons that can be identified for 
Germany’s reluctance (see Hartong, 2019; Hartong et al., 2020). One is the German 
federal constitution, as states hold almost all responsibility for educational governance 
and use bodies, such as the Standing Conference of the Education Ministers of the 
German States (KMK), to defend this authority against the Federal Ministry of 
Education (BMBF). This also includes the centralization of data. Simultaneously, 
German federalism is characterized by a distinction between inner and outer school 
responsibilities, with the states being responsible for everything related to the content 
of schooling (including hiring and paying teachers, curriculum development, mon- 
itoring) and local communities being responsible for school buildings, administrative 
staff, and technical infrastructure (Hartong et al., 2020). To date, this distinction is 
reflected in the lack of links between school administration systems, statistical data, 
monitoring systems, and performance data. 

After a shocking performance in the Programme for International Student 
Assessment (PISA) in 2001, German policymakers announced a “turn” towards 
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data-based, output-oriented governance (including the implementation of 
national education standards and standardized testing), followed by broad initia- 
tives to expand, better coordinate, and centralize both administrative and peda- 
gogical data, including performance data (Lange et al., 2014). Part of these 
initiatives was the KMK Core Data Set resolution in 2003, in which — for the 
first time in German history — the states agreed to record a defined amount of 
nationally standardized, individualized data (including school, student, and teacher 
data, yet excluding test performance data). In subsequent years this initiative 
strongly pushed the infrastructuring of educational data across policy spaces within 
the German federation. At the same time, and as mentioned earlier, the KDS did 
not nationally centralize the data collection, but rather all disaggregated data 
remained within the individual state data systems. Still, however, there was a 
significant increase of interoperability because state data was now (supposed to be) 
collected in the same way. A centrally assigned student identification number — 
which was discussed in the beginning stage of the KDS implementation — was 
abandoned due to heavy protest from many states, teachers, and students. Instead, 
numerous states introduced their own student identification systems. 

During that reform period, the influence of context also became clearly visible 
regarding a strong traditional skepticism against the marketization and for-profit 
actors in the governance of education. Hence, within emerging data infra- 
structures it was often research and public institutions (sometimes even established 
for that purpose) that became responsible for (and potentially then contracted out 
elements of) the development of testing, reporting, and data management tools. 
To date, it has remained significantly more difficult for the EdTech industry to 
build up or directly enter the datafication and digitalization market in Germany, 
compared with other countries around the world (Hartong & Forschler, 2019). 

Finally, the German context can be characterized as a system of rather high 
teacher professionalism and autonomy, long governed through a combination of 
extensive professional training and the state provision of “input” resources. As a 
consequence, skepticism against any kind of high-stakes data, standardized or 
external testing, and ranking of schools has remained high, which also means 
teacher (but also student) data related to any performance measurement are highly 
protected. 

While all these factors deeply affected the implementation and operation of 
data infrastructures and restricted data interoperability, policy developments also 
show how these de facto factors became increasingly confronted with the cen- 
tralizing and standardizing, often self-reinforcing, dynamics of data infrastructur- 
ing (Hartong & Forschler, 2019), as well as with a growing number of reform 
promoters, particularly with the rise of the “digital agenda” after 2012 (Forschler, 
2018). Since 2012, there has been a remarkable explosion of new intermediary 
policy networks, which brought together actors from various sectors and levels of 
policy, either directly (e.g., Bitkom, Digital Education Pact) or more indirectly (e. 
g., Alliance for Education/BfB, Forum Education Digitalization/fbd), interwoven 
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with EdTech interests. All of them characterized Germany as dramatically lagging 
globally, regarding either digital education in general, or data infrastructures and 
interoperability in particular, as well as hanging on to an outdated federal policy 
architecture. Simultaneously, actors such as Dataport (see Dataport, n.d.) or the IT 
planning council’ (see IT-Planungsrat, 2020), increased their investments in cross- 
state collaboration, data system transfer, or harmonization, as well as in coordinating 
the work of statistical offices at state and national levels (Hartong, 2019). This to a 
growing extent fostered cross-state (not supra-state) data infrastructuring, while 
bypassing the “old” political conflict. In various states, this also included multiple 
data security law adaptations, which explicitly allowed individual data to leave the 
schools, but not the states, if anonymized. 

In 2016, both the BMBF as well as the KMK responded to these shifting policies by 
politically cementing a digital agenda, which heavily supported data infrastructuring, 
centralization and standardization. Pushed by the BMBF and KMK initiatives, 2017 
and 2018 saw extensive digitalization, including the implementation of numerous 
digital education platforms that increasingly use single sign-on (SSO) identification 
tools. Such products not only envision the digital classroom, but also facilitate the 
linking of data generated via learning tools with other monitoring data, such as school 
information systems, test data, and school statistics (Hartong, 2018). 

While it remains to be seen how these developments will further change the 
German education context, the policy transformations thus far point to the ambivalent 
role data infrastructuring has played, ranging from contextual restriction/mediation to 
infrastructure-driven change (disruptive potential) of educational (policy) context. The 
following section is devoted to a better understanding of this ambivalence, using three 
different examples. 


Understanding the “Disruptive” Potential of Data Infrastructures: 
Three Examples from Germany 


The three examples illustrated in this section point to data infrastructures that 
emerged for distinct purposes (for example, data to inform policymaking, mon- 
itoring of schools, and/or improvement of classroom practice). These reflect the 
influence of the contextual conditions of possibility mentioned earlier in this 
chapter, but simultaneously the substantial de-/re-contextualizing dynamics set 
free by the processes of infrastructuring.” 


The Transforming Infrastructures of Standardized Assessments 


After the turn to the 21st century, two “nationally” standardized assessments were 
introduced in Germany: (1) the sample-based Bildungstrend to compare aggregated 
performance data of the states and their subsequent fulfillment of the national 
standards set in relation to the reforms, and (2) the census-based Vergleichsarbeiten 
(VERA) to improve classroom practices, based on the measurement of individual 
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students’ performance, which was again oriented towards the fulfilment of national 
standards. 

Both assessments originated from different contexts. The Bildungstrend had its 
roots in an oversampled PISA study to compare the states’ performances in the 
assessment, but after 2006 it was transformed and re-designed to monitor the 
states’ educational standards achievements and was then institutionally transferred 
to the newly established Institute for Educational Quality Improvement (Institut 
zur Qualitatsentwicklung im Bildungswesen [IQB]), under the supervision of the 
KMK. Still, the Bildungstrend was supposed to generate meta-knowledge about 
states’ average performance and thus inform national policy and overall system 
monitoring. 

In contrast, VERA originated from a test initiative by seven German states in 
2004, coordinated by a university in southern Germany but used individually and 
very differently in each of the states to support teachers in improving classroom 
practices. VERA was not used for accountability or for policy development. 
Triggered by the emerging national reform agenda and the monitoring strategy of 
the KMK, however, both assessments became integrated officially into one 
coherent testing infrastructure, obligatory for all states and aligned with the 
national education standards, with the IQB taking over coordination (Hartong, 
2018). This is an example of the way in which changing infrastructure alters the 
policy and governing context of schooling, with the IQB now not only produ- 
cing extensive datapools between local, state, national, and also international 
contexts, but also aligning these different test contexts and contents into one 
coherent, standardized (and in that sense interoperable) meta-frame. 

Still, however, the testing infrastructure remained significantly restricted, which is 
clearly visible when looking at the different parts of the infrastructuring process of 
test data. While the IQB is completely responsible for developing, administering, 
and reporting the Bildungstrend, VERA has taken the lead in developing, piloting, 
and scaling the tests (which also is an important practice of valuation), as well as 
providing standards-aligned learning tasks for teachers to support test preparation 
and test-related skill achievement. The states have remained responsible for 
administering VERA and for mediating the test data between schools and admin- 
istration individually, including optional adaptations and modularization of the test 
due to schools’ individual needs (see IQB, n.d.-a). Additionally, no individual test 
data are reported back to the IQB, but only aggregated data samples for test pilot- 
ing and further development. The states are required to refrain from grading the 
test or publishing and publicly ranking the test data of individual schools (KMK, 
2018). As a result of these restrictions, VERA not only still varies considerably 
between the states, but there is also a deliberate waiver of interoperability regarding 
the links between test data and grading, public platforms and also supra-state col- 
lection of individual test data (see Hartong & Piattoeva, 2019). Nevertheless, there 
are still significant consequences of the ongoing VERA transformations, which 
refer to the gradual centralization/expansion of testing into/within the IQB (lately 
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also including the development of nationally standardized school graduation tasks, 
see IQB, n.d.-b). These transformations have made state test data gradually more, 
though not fully, interoperable. It remains to be seen how this will change further 
with VERA now gradually transforming into an online assessment (for example, 
Institut ftir Schulqualitét der Lander Berlin und Brandenburg e.V. [ISQ-BB], for 
the states Berlin and Brandenburg, see ISQ-BB, n.d.) and some states already 
grading and reporting the test results via digital online portals (e.g., Zentrum ftir 
Empirische Padagogische Forschung [zepf], see zepf, 2020). At the same time, there 
is also growing data interoperability within the individual states, namely between 
VERA data collected in the schools/for classroom development and the data sys- 
tems used in the state agencies for monitoring school performance and develop- 
ment, explicitly made possible with the KMK’s VERA resolution of 2018. 


The Transforming Infrastructures of State School Monitoring 


With the rise of the national education reforms, German state-level education 
agencies (Bildungsbehérden auf Bundeslandebene) have been urged to produce 
growing amounts of nationally standardized data (including the KDS and the 
aforementioned tests) and to use that data for more effective and efficient school 
leadership and monitoring. Consequently, over the past decade, investments in 
data infrastructures and interoperability have significantly increased, which 
includes data transfer between individual schools and state agencies, but also 
includes data transfer within different state agencies’ institutions, departments, or 
sectors.” 

Again, the German states approached data system transformations differently. 
States opted early on for either a centralized solution, in which data from all 
schools within the state were collected within the same data systems, or they held 
on to a decentralized solution, in which school data systems vary but a growing 
amount of standardized data reports are required to be submitted to centralized 
state databases (Hartong et al., 2020, p. 7). Unsurprisingly, the number of states 
choosing a centralized (though potentially voluntary) solution has grown, and this 
trend seems likely to continue in the future. This is also due to the fact that in 
many cases, states have either joined forces and bought (or received free-of- 
charge) systems from other states, or they have partnered with external providers 
that produce solutions for multiple states. Some of the many examples of this are 
Hamburg, which adopted and further developed Brandenburg’s school manage- 
ment software WeBBSchule, and Bavaria and Baden-Wuerttemberg, which each 
developed its system in collaboration with the private provider ISB AG (which 
hosts the system edoo.sys; ISB AG, 2020). 

At the same time, as mentioned above, in many states, there has remained a 
significant infrastructural gap between the organization of performance data — 
often exercised by newly founded quality and/or school monitoring institutes 
(Rürup, 2018) — and the more traditional organization of school statistics and 
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resource planning. Often this occurs in concert with the statistical institutes of the 
states, which serve more sectors than education. For example, in Hamburg, most 
education data within the state school agency are centrally stored within a data 
warehouse, and data generated by the quality institute are still organized within a 
separated database. For the quality institute’s work with statistical data, it receives 
selected data cubes from the state agency. Thus, the fragmentary nature of many 
school monitoring infrastructures (see Breiter & Lange, 2019) goes well beyond 
the gap between statistical and performance data. 

Despite this fragmentation, there has been a clear shift towards data-based 
school supervision and consultancy practices, such as setting target agreements 
with principals for school development. Data used for this purpose range from 
financial and resource information, to aggregated test or graduation exam data 
and class cancellations. Different from the IQB that aligns tests nationally (see 
previous section), data are usually reported separately, often based on distinct data 
formats, time frames, or meta-data, even though some states (such as Hamburg, 
Brandenburg, and Bremen) try to integrate them into “at a glance” school data 
profiles. As mentioned previously, the German system is also characterized by a 
rather high skepticism towards high stakes testing data, which is especially true in 
relation to the usage of such data for holding individual teachers accountable. For 
instance, individual class or student VERA data should not be included in school 
supervision’, but also unfulfilled target agreements or performance measurements 
in most cases do not lead to any formalized “hard” consequences for the schools 
(such as in the United States; for Berlin, see Baur, 2016). Importantly, this does 
not mean that the context of school supervision and consultancy has not become, 
or is increasingly becoming, affected by the ongoing infrastructuring dynamics, 
but rather that it seems important to closely trace how and where it is occurring.” 
Hereby, a key role is apparently played by a growing number of anonymization 
initiatives (in Germany, it is referred to as “pseudonymization”), which in many 
states have made it possible to process individual (particularly longitudinal) student 
and teacher data beyond the school level. Additionally, as recent debates on the 
implementation of a national education register indicate, transformations may lead 
Germany to nationally centralized individual databases (Fickermann & Weishaupt, 
2019). 


The Transforming Infrastructure of School Platforms and Learning 
Management Systems 


The third example that illuminates the ongoing dynamics, as well as the dis- 
ruptive potential of data infrastructuring in Germany, is the rise of school plat- 
forms and learning management systems (LMS).° Driven by the goal to support 
digital learning and teaching, while simultaneously making school data manage- 
ment more efficient, such platforms or LMSs often combine features such as file 
storage, communication interfaces, calendars, curriculum development tools, 
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learning analytics and grading management, classroom organization, school admin- 
istration, and human resource management. This means that a context-sensitive 
approach needs to consider the included features of a particular platform or system, 
how it is used by different users, and how it relates to other kinds of data 
infrastructures. 

While school platforms or LMSs are still less common in Germany than in many 
other countries, their implementation significantly increased with the rise of the 
digital agenda after 2012, particularly with the national digitalization initiatives of 
2016. As a result, a growing number of schools have implemented some kind of 
LMS, either individually or as part of a city- or state-led initiative. Again, there are 
substantial differences between the states, meaning that they each offer different 
market conditions for vendors. Some states developed one “ready-to-use” LMS 
(either through in-house initiatives or through the buy-in of vendor products), and 
they actively encourage schools to use that standardized technology. At the same 
time, even though particular vendors operate within different state markets (such as 
the platform itslearning), there is, at least so far, no cross-state LMS or platform 
operation. This disparity recently caused the federal government to initiate a national 
school platform solution (named School Cloud) by the SAP-associated (SAP, n.d.) 
research institute Hasso-Plattner-Institut (HPI; see HPI, 2020). While this initiative is 
still in its infancy, it recently gained tremendous momentum with COVID-19 
school closures. 

Simultaneously, and in response to this competing national platform initiative, 
mid-sized LMS providers increased their efforts and gained substantial market 
influence in Germany, such as itslearning. Originating from a Norwegian initia- 
tive in 1998, itslearning has become one of the leading LMS providers in Europe 
and beyond, until it was recently sold to the global investment organization 
Sanoma (Sanoma, 2019)). Itslearning is a web-based platform for learning mate- 
rials, grading, curriculum design, communication, and school administration, 
while simultaneously offering multiple interfaces to applications, including ency- 
clopedias, online-tutoring, and subject-specific learning tools. At the same time, 
itslearning actively promotes data interoperability among different stakeholders, 
including automated data exchange across school, parent, and state agency data- 
bases, but also features classroom data for monitoring and reporting purposes 
(itslearning, 2020). 

In Germany, itslearning gained particular attention as part of a public-private 
partnership in the state of Bremen, where the state education agency not only 
made the platform available to all its schools, but also integrated itslearning into 
standard preparatory training for principals at the Bremen Institute for Teacher 
Training. The platform receives individual student and teacher data from the 
central state school monitoring system to more easily create classes, courses or to 
assign tasks to students.’ At the same time, the vendor and the Bremen school 
consultancy agency (Institute for Schools [LIS]) worked together to refine and 
adapt the platform to the German/Bremen context. Importantly, this included 
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prohibiting (technically deactivating) the processing of teachers’ and students’ 
log data and also the usage of itslearning data for any kind of supra-school 
monitoring. 

Additionally, despite its state-wide application, schools in Bremen still widely 
differ in how they use itslearning. While some schools and individual teachers use 
the platform primarily as a file storage and communication tool, others transferred 
significant amounts of pedagogical tasks to the platform. Interestingly, as with the 
dynamics described previously in this chapter, the more schools work with 
itslearning, the more it seems to stimulate more extensive usage, digital cen- 
tralization (of administrative and pedagogical/performance data), and the elim- 
ination of alternate products. 


Conclusion and Outlook 


As the three examples show, Germany offers an interesting case to trace both the 
importance of (restricting) conditions of possibility when seeking to understand 
data infrastructures and interoperability, and the actual power, or the disruptive 
potential, of data infrastructures for education policy, governance, and practice 
over time. As argued at the beginning of this chapter, this potential derives mostly 
from ongoing standardization (as visible in the alignment of state school mon- 
itoring infrastructure among states) and centralization (as visible in the VERA 
testing infrastructures or in the case of itslearning). Both include significant shifts 
of discourse, actor constellations, but also substantial decontextualization and 
recontextualization regarding which data are collected, how, by whom, and for 
what purpose. While the German context might, in many ways, still thwart the 
infrastructuring of data a lot more than is the case in other countries, in Germany 
such discourse and actor shifts as well as de- and recontextualization processes are 
already clearly visible. As an example, while the implementation of a national 
student identification number was heavily pushed back 15 years ago, more recent 
debates on a national student register go almost unnoticed by the opponents of data 
centralization. One important reason lies in the aforementioned shift towards 
“pseudonymization” as a working solution to protect individual student data, while 
still making these data usable for policymaking or research (see Fickermann & 
Weishaupt, 2019).° Here, Germany mirrors global discourses on data infrastructure 
and interoperability, which today have often either shifted towards questions of 
(deficient) functionality or towards requesting “good” data security laws (often 
named “safe data”), the fulfillment of ethical considerations or the alignment of data 
(collection/processing) to standards (see Wyatt-Smith et al., 2019). 

While we do not argue that legal and ethical issues are not extremely impor- 
tant, particularly given that data misuse or abuse is still prevalent, our point is that 
a critical perspective on data infrastructures and interoperability should not focus 
on these issues alone. Put differently, important questions could be neglected as 
soon as the focus is put solely on the definition of standards, ethical checklists, or 
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data ownership — all of which are not only limiting, but actually also further push 
(because they legitimize a particular way of) data infrastructuring. For example, 
what is the actual difference between individual data being collected anon- 
ymously or non-anonymously within a nationally centralized database, when in 
both cases these data cause “real” consequences for policy and governance? Why 
could it be problematic to have all data collection (done either by public or pri- 
vate actors) following the same standard instead of provoking disagreement, ten- 
sion — or even critical reflection — on rules of data in-formation and, thus, on the 
ongoing politics of data and data infrastructures? 

Consequently, if we take these kinds of questions seriously, a major implication 
for action lies in strengthening the awareness and critical understanding of data 
infrastructuring as the emergence of “educational settings” (Decuypere, 2019), 
which to a growing extent “in-form” what is made visible and acted upon as 
schooling. In other words, only if we better understand why such infrastructures 
are educational, or disciplinary, on whom and how (and here is a pressing need 
for further research), the modes and effects of their disruption can be grasped, 
problematized, and ultimately regulated. This does not mean that infrastructures 
form fixed modes of governance. As the case of Germany shows, data infra- 
structures reveal an ongoing dynamic, while always being dependent on their 
actual enactment, for example, by people using the data. At the same time, as the 
illuminated examples made clear, even for the latecomer Germany, it is this 
ongoing, self-reinforcing dynamic that warrants sustained global attention. 
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Notes 


1 Even though it is not specific to the field of education, a major federal reform in 2009 
was the newly inserted Article 91c of the Basic Law, which heavily promotes infrastructure 
of administrative data. It promotes, “the legal prerequisites for seamless electronic com- 
munication among federal, state, and local government agencies” (IT Planning Council, 
2020, para. 2). To implement this article, the federal and state governments adopted the 
State IT Treaty, which also included the establishment of the IT Planning Council as the 
body responsible for information technology cooperation in Germany. 

2 The findings illuminated in this chapter derive from research data generated and ana- 
lyzed by the authors from a range of previous and ongoing research projects. All pro- 
jects are built on the analysis of policy documents, laws and ministerial circulars, website 
information, interviews with policy actors, and school officials, as well as (in the case of 
itslearning) participatory observations. For more detailed methodological explanations, 
see Datafied (2020), Hartong and Förschler (2019), and Hartong and Piattoeva (2019). 
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3 This for example includes interfaces between the local resident registries and school 
information system to monitor student numbers and school attendance, as in Hamburg 
or Bremen. 

4 Yet allowing it at all to be used for supervision (and thus accountability) purposes can 
already be interpreted as an effect of data infrastructuring. 

5 In four German states, this is currently analyzed in the Datafied project (see Datafied, 
2020). One example of tracing these dynamics is the transformation of social indices for 
schools (Hartong & Breiter, in press). 

6 For the purpose of this chapter, we use the terms LMS and learning platform inter- 
changeably to refer to a particular type of data infrastructure. We simultaneously 
acknowledge the challenge of clear definitions, which also applies to School Manage- 
ment Information Systems (SMIS), Learning Content Management Systems (LCMS), 
and School Clouds (Breiter & Lange, 2019; Hughes & Attwell, 2009). 

7 A similar development can be observed in Hamburg, where itslearning has become 
linked to the centralized state school portal eduPort. 

8 The infrastructure of research data has been an often-underestimated driving force in 
the overall emergence of data infrastructure in schools. For example, and similar to the 
OECD, the IQB acquired an enormous database of student performance data and 
background information. This database is further integrated with other datasets (e.g., 
PISA) within an IQB in-house research database, which makes it available for re-analysis 
and secondary analysis. 


References 


Baur, C. (2016). Merkmalsbezogene ressourcenausstattung von schulen in Berlin - das bonus- 
programm zur sntersttitzung von schulen in schwieriger lage. Die Deutsche Schule, 108(4), 
370. https://www.waxmann.com/index.php?eID=download&id_artikel=ART102058& 
uid=frei. 

Breiter, A., & Lange, A. (2019). Die digitale schule und schulverwaltung. In H. H. Lühr, 
R. Jabkowski & S. Smentek (Eds.), Handbuch digitale verwaltung (pp. 330-342). Kom- 
munal-und Schul-Verlag (KSV). 

Bündnis ftir Bildung. (2019). Interoperabilitaet & datenschutz. https://www.bfb.org/ 
interoperabilitaet-und-datenschutz. 

Datafied. (2020). Homepage. https://www.datafied.de. 

Dataport. (n.d.). Wir machen digitale zukunft. https://www.dataport.de. 

Decuypere, M. (2019). Researching educational apps: Ecologies, technologies, sub- 
jectivities and learning regimes. Learning, Media and Technology, 44(4), 414-429. http:// 
doi.org/10.1080/17439884.2019.1667824. 

Ed-Fi Alliance, LLC. (2020). Homepage. https://www.ed-fi.org. 

Fickermann, D., & Weishaupt, H. (Eds.). (2019). Bildungsforschung mit daten der amtlichen 
statistik. Die Deutsche schule, 14. Waxmann. https://doi.org/10.31244/dds.bh.2019.14. 

Forschler, A. (2018). Das “who is who?” der Deutschen bildungs-digitalisierungsagenda — 
eine kritische politiknetzwerk-analyse. Pädagogische Korrespondenz, 58(2), 31-52. 

Gulson, K., & Sellar, S. (2018). Emerging data infrastructures and the new topologies of 
education policy. Environment and Planning D: Society and Space, 37(2), 350-366. https:// 
doi.org/10.1177/0263775818813144. 

Hartong, S. (2016). Between assessments, digital technologies and big data: The growing 
influence of “hidden” data mediators in education. European Educational Research Journal, 


15(5), 523-536. https://doi.org/10.1177/14749041 16648966. 


Data Infrastructures and Interoperability 149 


Hartong, S. (2018). Towards a topological re-assemblage of education policy? Observing 
the implementation of performance data infrastructures and “centers of calculation” in 
Germany. Globalisation, Societies and Education, 16(1), 134-150. https://doi.org/10.1080/ 
14767724.2017.1390665. 

Hartong, S. (2019). The transformation of state monitoring systems in Germany and the 
US: Relating the datafication and digitalization of education to the global education 
industry. In M. Parreira do Amaral, G. Steiner-Khamsi, & C. Thompson (Eds.), 
Researching the global education industry — Commodification, the market and business involvement 
(pp. 157-180). Palgrave Macmillan. https://doi.org/10.1007/978-3-030-04236-3_8. 

Hartong, S. (2020). The power of relation-making: insights into the production and 
operation of digital school performance platforms in the US. Critical Studies in Education. 
https://doi.org/10.1080/17508487.2020.1749861. 

Hartong, S., & Breiter, A. (in press). Between fairness optimization and “inequalities of 
dataveillance” — the emergence and transformation of social indices in German school 
monitoring and management. In A. Verger, C. Maroy & S. Grek (Eds.), World Yearbook 
of Education 2021. 

Hartong, S., Breiter, A., Jarke, J., & Forschler, A. (2020). Digitalisierung der staatlichen 
schulverwaltung. In T. Klenk, F. Nullmeier, & G. Wewer (Eds.), Handbuch staat und 
verwaltung im digitalen zeitalter. Springer. 

Hartong, S., & Förschler, A. (2019). Opening the black box of data-based school mon- 
itoring: Data infrastructures, flows and practices in state education agencies. Big Data & 
Society, 6(1), 1—12. https://doi.org/10.1177/2053951719853311. 

Hartong, S., & Piattoeva, N. (2019). Contextualizing the datafication of schooling: A 
comparative discussion of Germany and Russia. Critical Studies in Education. https://doi. 
org/10.1080/17508487.2019.1618887. 

Hartong, S., Piattoeva, N., Saari, A., & Savage, G. (in press). Transformations of 
education policy and governance in the digital era: Cross-country reflections. In M. 
Busemeyer, P. Marx, A. Kemmerling, & K. van Kersbergen (Eds.), Digitalization and 
the welfare state. 

Healthcare Information and Management Systems Society, Inc. (2020). Interoperability in 
the healthcare ecosystem. https://www.himss.org/what-interoperability. 

Hogan, A., Sellar, S., & Lingard, B. (2016). Commercialising comparison: Pearson puts the 
TLC in soft capitalism. Journal of Education Policy, 31(3), 243—258. https://doi.org/10. 
1080/02680939.2015.1112922. 

HPI. (2020). The HPI-Schul-Cloud. https://hpi.de/en/open-campus/hpi-initiatives/hpi- 
schul-cloud.html. 

Hughes, J., & Attwell, G. (2009). TACCLE. Teachers’ aids on creating content for learning 
environments: The e-learning handbook for classroom teachers. TACCLE. http://taccle.eu/ 
wordpress/wpcontent/uploads/2013/09/TACCLEEnglishDEF . pdf. 

IQB.. (n.d.-a). VERA — Ein tiberblick. https://www.igb.hu-berlin.de/vera. 

IQB. (n.d.-b). Gemeinsame abituraufgabenpools der lander. www.iqb.hu-berlin.de/abitur. 

ISB AG. (2020). edoo.sys — Schulverwaltung leicht gemacht. https://www.edoo-sys.de. 

ISQ-BB. (n.d.). VERA 8 online 2020. http://www.isq-bb.de/wordpress/vera-online/. 

IT Planning Council. (2020). State treaty on IT. www. it-planungsrat.de/EN/it-planing- 
council/RechtlicheGrundlagen/rechtliche_grundlagen_node.html. 

IT-Planungsrat. (2020). Homepage. https://www.itplanungsrat.de/DE/Home/home_ 
node.html. 

itslearning. (2020). Features. https://itslearning.com/global/k-12/features/. 


150 Hartong, Forschler, and Dabisch 


Kauko, J., Rinne, R., & Takala, T. (Eds.). (2018). Politics of quality in education: A com- 
parative study of Brazil, China, and Russia. Routledge. https://doi.org/10.4324/ 
9780203712306. 

KMK. (2018). Vereinbarung zur weiterentwicklung der vergleichsarbeiten (VERA). https:// 
www.kmk.org/fileadmin/Dateien/veroeftentlichungen_beschluesse/2012/2012_03_08_ 
Weiterentwicklung-VERA. pdf. 

Koyama, J. P. (2011). Generating, comparing, manipulating, categorizing: Reporting, and 
sometimes fabricating data to comply with No Child Left Behind mandates. Journal of 
Education Policy, 26(5), 701-720. https://doi.org/10.1080/02680939.201 1.587542. 

Kubicek, H., Breiter, A., & Jarke, J. (2019). Daten, metadaten, interoperabilitat. In T. 
Klenk, F. Nullmeier & G. Wewer (Eds.), Handbuch digitalisierung in staat und verwaltung. 
Springer VS. https://doi.org/10.1007/978-3-658-23669-4_1-1. 

Lange, A., Gronert, T., & Breiter, A. (2014). Schulverwaltungssoftware in den bunde- 
slandern 2014. https://www.ifib.de/publikationsdateien/SVS_in_den_Bundeslindern_ 
2014 _Final_.pdf: 

Lingard, B. (2019). The global education industry, data infrastructures, and the restructur- 
ing of government school systems. In M. Parreira do Amaral, G. Steiner-Khamsi, & C. 
Thompson (Eds.), Researching the global education industry — Commodification, the market and 
business involvement (pp. 135-155). Palgrave Macmillan. https://doi.org/10.1007/ 
978-3-030-04236-3_7. 

Mau, S. (2019). The metric society: On the quantification of the social. John Wiley & Sons. 

Pangrazio, L. (2019). What is data interoperability and why is it important in education? Data 
Smart Schools. https://data-smart-schools.net/2019/08/01/what-is-data-interoperability- 
and-why-is-it-important-in-education/. 

Ruppert, E. (2012). The governmental topologies of database devices. Theory, Culture and 
Society, 29(4-5), 116-136. https://doi.org/10.1177/0263276412439428. 

Rürup, M. (2018). Berichtet bildungsberichterstattung über bildung? Eine ausei- 
nandersetzung mit kritikerinnen. In I. Bormann, S. Hartong, & T. Höhne (Eds.), Bildung 
unter beobachtung: Kritische perspektiven auf bildungsberichterstattung (pp. 16—42). Juventa. 

Sanoma. (2019). Sanoma acquires itslearning, an international learning platform provider. 
https://sanoma.com/release/sanoma-acquires-itslearning-an-international-learning-pla 
tform-provider/. 

SAP. (n.d.). What is SAP?https://www.sap.com/corporate/en.html. 

Savage, G. C. (2019). What is policy assemblage? Territory, Politics, Governance, 8(3), 319— 
335. https://doi.org/10.1080/21622671.2018.1559760. 

Sellar, S. (2015). Data infrastructure: A review of expanding accountability systems and 
large-scale assessments in education. Discourse: Studies in the Cultural Politics of Education, 
36(5), 765-777. https://doi.org/10.1080/01596306.2014.931117. 

Sellar, S. (2017). Making network markets in education: The development of data infra- 
structure in Australian schooling. Globalisation, Societies and Education, 15(3), 341-351. 
https://doi.org/10.1080/14767724.2017.1330137. 

SIF Association. (n.d.) Access 4 Learning Community. https://www.a4l.org. 

Takayama, K., & Lingard, B. (2019). Datafication of schooling in Japan. Journal of Education 
Policy, 34(4), 1-21. https://doi.org/10.1080/02680939.2018.1518542. 

US Department of Education. (2020). Digital systems interoperability. https://www.ed. 
gov/open/plan/digital-systems-interoperability. 

Williamson, B. (2016). Boundary brokers: Mobile policy networks, database pedagogies, 
and algorithmic governance in education. In T. Ryberg, C. Sinclair, S. Bayne & M. de 


Data Infrastructures and Interoperability 151 


Laat (Eds.), Research, boundaries, and policy in networked learning (pp. 41-57). Springer. https:// 
doi.org/10.1007/978-3-319-31130-2_3. 

Wyatt-Smith, C., Lingard, B., & Heck, E. (2019). Digital learning assessments and big data: 
Implications for teacher professionalism. UNESCO Working Papers. https://unesdoc.unesco. 
org/ark:/48223/pf0000370940/PDF/370940eng.pdf.multi. 

zepf. (2020). VERA 3. https://www.projekt-vera3.de/vera/vw_login.php. 


9 


DATAFICATION AND SURVEILLANCE 
CAPITALISM 


The Texas Teacher Evaluation and Support 
System (T-TESS) 


Jessica Holloway and Steven Lewis 


Introduction 


The election of Donald Trump as president of the United States (US) in 
November 2016 will likely be remembered as one of the greatest upsets in 
modern political history. What was thought to be a sure win for candidate Hillary 
Clinton quickly turned into a shocking defeat, leaving political experts rattled and 
searching for new analytical tools to help explain their miscalculations. More 
recently, details have emerged about the (previously) little-known data analytics 
company Cambridge Analytica and its role — and that of data more generally — in 
securing the Trump Presidency, reflecting the increasing power that big data and 
technical analytics exert over the daily lives of citizens. What the public learned 
was that their seemingly private participation on various websites (such as Face- 
book) was being tracked and collected, often without their knowledge or express 
consent. These data were then “mined,”' sold, and analyzed for profiling poten- 
tial demographic and political factions within the US electorate, meaning people 
were targeted for political grooming via specific advertising and content. 
Although polling techniques and political targeting are constant and consistent 
features of modern politics, what made Cambridge Analytica so unique was its 
purposeful disregard for the human candidate. While traditional campaign tech- 
niques might start with a candidate’s policy position and then use polling data to 
craft a targeted message for a particular constituency, Cambridge Analytica started 
with the data and instead worked backwards. The company first used data mined 
from social media sites to identify the “mood” of constituents, before testing 
various emotive messages to craft a policy position. Relying on years of harvested 
data and message testing, Cambridge Analytica helped manufacture the perfect 
primed candidate for any given electorate. The candidate needed not be 
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ideologically sound, socially pure, or experientially prepared, but merely “pro- 
grammable”; that is, capable of being encoded with tried and tested scripts, be 
they “Make America Great Again,” “Build the Wall,” or “Lock Her Up.” 

Under these circumstances, the government outsider, Donald Trump, cannot 
alone be credited for orchestrating the defeat of Hillary Clinton and the political 
establishment writ large. Rather, he was in many ways the vessel for a broader 
resurgence of reactionary nativism, parochialism, and protectionism, albeit tem- 
pered with the predictive power of contemporary big data and technical analytics. 
Indeed, then-Chief Executive Officer of Cambridge Analytica, Alexander Nix, 
was recorded observing that “they’re politicians, they’re not technical. They don’t 
understand how it [data analytics] works” (McKee, 2018, para. 12). In this brave 
new world of datafication, it is the (exceedingly private) world of software engi- 
neers, data miners, and program developers who steer campaigns, forge candi- 
dates, and shape public opinion. 

While the public grapples with the unprecedented and unforeseen influence of 
“big data” and analytic techniques in recent campaign and election processes, 
other public domains are similarly being subjected to the datafication of practices, 
services, and people. Education, for example, has been particularly reconceptua- 
lized by digital data, as well as the technicians and statisticians that produce, ana- 
lyze, and disseminate such data (Williamson, 2016a, 2016b). In this chapter, we 
look to the schooling system in Texas (US) to illustrate how schools and teachers 
are becoming sites for data collection that can potentially feed the growing 
market of what Zuboff (2019a) called “surveillance capitalism.” We analyze the 
public-private partnerships (PPPs) between Texas public schools and three orga- 
nizations: SAS Institute Inc., Responsive Learning, and the National Institute for 
Excellence in Teaching (NIET). In collaboration with the Texas Education 
Agency (TEA), these groups have built the Texas Teacher Evaluation Support 
System (T-TESS), which is used state-wide to manage a broad array of schooling 
processes. T-TESS is an ensemble of digital services, applications, and analytic 
reporting that relies on various modes of statistical calculation, prediction, and 
computational processing to monitor and evaluate teachers. Our purpose in this 
chapter is two-fold: first, we identify the roles, techniques, and features of the 
partners’ contributions to the T-TESS, and second, we use contemporary think- 
ing around “datafication” (Bradbury, 2019; Williamson, 2015) and “surveillance 
capitalism” (Zuboff, 2019a) to examine how these actors contribute to, as well as 
stand to benefit from, their partnerships with schools in Texas. 

Specifically, we will look at how the T-TESS platform and evaluation system 
has enabled external organizations to establish prominent positions in the Texas 
education system. The varied expertise, capabilities, and strategic efforts of these 
organizations have secured them roles as key providers of evaluation, professional 
development, and data-related support. This has led to, for example, revenue 
incentives and streams for non-public institutions, and the ability for private actors 
to gain access to data that can be used for private, business-oriented purposes. In 
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concert with the intensifying datafication of education, and the abundance of 
personal data through interactions with software platforms such as the T-TESS, 
we suggest that schooling, and the teaching profession itself, are becoming 
increasingly positioned as fertile ground for the unleashing of surveillance 
capitalism. 


The Datafication of Teachers and Teaching: Providing Fertile 
Ground for Surveillance Capitalism 


In recent years, there has been a significant increase in data collection and use in 
education globally, reflecting the relevance of numbers (Ozga, 2016), data 
(Selwyn, 2015), data visualizations (Williamson, 2016b), and software platforms 
(Decuypere & Landri, 2020; Lewis, 2020) for purposes of measuring, comparing, 
and governing school performance. Powerful trends in educational policy to 
establish high stakes teacher accountability systems have led testing, evaluation, 
and dis/incentivization to become key drivers that shape teacher practice and 
define teacher “quality” (Bradbury & Roberts-Holmes, 2017; Holloway et al., 
2017; Lingard, 2010), informing such significant considerations as performance 
pay, promotion, and retention. There is now a growing consensus within edu- 
cation literature that this form of high stakes accountability undermines teacher 
expertise, authority, and professionalism; constrains the capacity of teachers to act 
on their own professional knowledge, experience, and discretion; and risks 
endangering reputations, career progression, and jobs (Hardy, 2018; Holloway, 
2014; Thompson & Cook, 2014). 

Indeed, the notion of teachers being deemed “risky” subjects (Hardy, 2015; 
Holloway, 2019) has itself become dominant within high stakes accountability 
regimes and political rhetoric, with teachers discursively positioned as potential 
impediments to future student performance and national economic prosperity, 
and in need of policy interventions that might mitigate such risks. Following such 
logic, compliance-based teacher accountability has become increasingly prevalent, 
and at the same time, particularly challenging to teacher professionalism, as modes 
of standardization, datafication, and evaluation narrow the possibilities for teachers 
to rely solely on their professional knowledge, training, and experience (Hardy, 
2018; Lewis & Holloway, 2019). Rather, teachers face increased pressure to 
employ numerical data (i.e., from standardized literacy and numeracy tests), eva- 
luative tools (e.g., observation rubrics), and prescriptive definitions of “what 
works” (Lewis, 2017a, 2020) to guide their pedagogical decisions and classroom 
practices (Bradbury & Roberts-Holmes, 2017). 

This emphasis on using (largely numerical) data to inform teacher practice and 
school policy is concurrent with moves towards what have been described as 
digital, or algorithmic, modes of governance (Williamson, 2015, 2016a, 2016b), 
reflecting the “apparent power, agential capacity and control that algorithms 
command of our lives” (Neyland, 2015, p. 199). Such developments have 
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accompanied the unprecedented growth and use of digital forms of data in edu- 
cation governance, expressed as increasing volume, velocity, and variety (the 
“three Vs”). This has resulted in school practices increasingly collapsing into data 
management practices, typified by what Selwyn et al. (2015) have tellingly 
described as teachers and principals having to curate “200,000 lines on an Excel 
spreadsheet” or “the biggest spreadsheet in the world” (p. 771), rather than focus 
their attention and energy on more educative considerations (such as pedagogy 
and curriculum). Central to understanding the logic that underpins algorithmic 
modes of governance, the power attributed to algorithms emerges through “non- 
numerical objects” (e.g., teachers and students) being rendered in numerical terms 
(such as, value-added scores and teacher observation scores), with algorithms 
recursively defining numerical outputs as the basis for subsequent inputs (Ney- 
land, 2015). Translating complex social domains and practices like schooling into 
numerical data to be iteratively fed into a centrally coordinating algorithm also, 
arguably, provides fertile opportunities for digital software platforms and their 
providers to occupy a significant role in policymaking and practice. 

As such, we find ourselves in a moment when the unprecedented need for, 
and generation of, performance data in education is drastically reshaping school- 
ing. Alongside demands for increased accountability and transparency in public 
schooling, these data have produced new urgencies around finding “evidence- 
informed” (Lingard, 2013) solutions to putative problems of policy and practice. 
This desire for “fast policy” solutions (Lewis & Hogan, 2019; Peck & Theodore, 
2015) and the increasing centrality of digital software platforms in school gov- 
ernance have produced a new market for policy populated by new service pro- 
viders, with efforts to identify “what works” occurring in tandem with the 
increased presence of non-government organizations in education. As such, 
powerful transnational private policy networks, which encompass intergovern- 
mental organizations, for-profit businesses, not-for-profit agencies, and the phi- 
lanthropic sector, now contribute to a global education industry that is reportedly 
worth more than US$4 trillion annually (Verger et al., 2016). 

As others have argued elsewhere, the involvement of private actors in public 
matters (like education) is problematic for a number of reasons, including the 
possible fundamental conflicts between private and public interests (see Lewis, 
2017b; Verger et al., 2016). However, recent developments have seen the focus 
of these concerns shift from the presence of private actors in public spaces, such as 
schooling (for example, for-profit edu-businesses providing services), to what is 
potentially now the wholesale collection and selling of private (as in personal) 
data by private organizations. In her book, The age of surveillance capitalism: The 
fight for a human future at the new frontier of power, Zuboff (2019a) develops this 
critique to suggest that private companies not only make money for the services 
they provide to public entities, but that it is the data these companies can collect 
via such arrangements that hold the real value. Here, the “primary” service pro- 
vided by the company serves as a vehicle through which the company can collect 
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user data, rather than the product or service being the primary end goal for the 
company. Zuboff (2019a) argues we are currently experiencing a massive shift 
towards surveillance capitalism, with the underlying focus being the raw data that 
companies (such as Google and Facebook) collect and then use for other business 
purposes. This is colloquially referred to in the technology industry as “data 
exhaust.” 

Importantly, such an approach dictates that all data are valuable, regardless of 
their original purpose (either actual or perceived by the user) or the means by 
which the data were collected. When big data analytics are applied to these 
massive data sets, inferences can then be drawn about all types of matters (e.g., 
political preferences, hobbies, religious affiliation), and this information can be 
used to inform targeted advertising. Zuboff (2019b) suggests that the evolution of 
capitalism involves the commodification of non-market products and practices. 
She further suggests that this is, 


how we turned making a living into “labor” and nature into “real estate.” 
Surveillance capitalism now claims private human experience as free raw 
material for translation into behavioral predictions that are bought and sold in 
a new kind of private marketplace. And it takes place almost completely 
without our knowledge. 


(para 4) 


While a given public consists of people from different geographic locations, 
demographics, political persuasions, and so forth, surveillance capitalists flatten 
these differences to see people simply as “users”; that is, generators of data via 
“private human experience” that can then be commodified and on-sold. This 
transforms the complexity of our (otherwise private) thoughts, desires, intentions, 
dreams, fears, histories, etc., into a digital dataset, the analysis of which enables 
these subjective human dispositions to be profitably identified and steered. 

It is also worth noting that surveillance capitalism is entirely dependent on a 
dual process that Zuboff (2019a) has termed “rendition,” whereby human 
experience is translated, so to speak, into a datafied version that is amenable to 
collection and analysis. The polysemic quality of the term “to render” is especially 
helpful here. On one hand, it describes how something is intentionally converted 
from one form to another; on the other, it means to deliver or surrender some- 
thing, often in an obliging way, as in the invocation to “render unto Caesar” in 
the Gospel of Saint Matthew (see Zuboff, 2019a, p. 234).” Taken together, we 
can see how surveillance capitalism requires rendering private human thoughts 
and actions into data (as datafication), as well as our active (and often consenting) 
giving of these data via engagement with technologies and software platforms that 
can perform these collections and translations. 

While we are admittedly yet to see the broad adoption of surveillance capital- 
ism in education specifically, recent trends in datafying teachers and teaching — 
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often for purposes of accountability — arguably position schooling as fertile 
ground for development by surveillance capitalists. This is especially so in the US, 
where the use of certain accountability tools and mechanisms, often by external 
providers, has historically been legislatively mandated or, at least, incentivized by 
federal grant initiatives (such as the Obama Presidency’s “Race to the Top” 
program). Given the already close relationship between schools in the US and 
private technology firms that frequently provide these data-related services, there 
is the risk that processes of datafication might, if unchecked, provide the ideal 
conditions for surveillance capitalism to emerge. In order to explore these possible 
concerns, we now turn our attention to the T-TESS, which is a specific example 
of a technology-based school accountability platform in the state of Texas. 


Teacher Accountability and the T-TESS 


Texas has a long and contentious history with test-based accountability. In fact, the 
No Child Left Behind (NCLB) Act has been largely credited to the Texas education 
system (and the Houston Independent School District, in particular), which was 
described by then-President George W. Bush as the “Texas Miracle” (Leung, 2004). 
Regarding teacher accountability, Texan teachers have faced high stakes con- 
sequences for decades, including (to varying degrees) termination for failing to “add” 
enough “value” to their students’ test scores on large-scale standardized assessments 
(Amrein-Beardsley & Collins, 2012). The state’s current teacher evaluation system is 
the Texas Teacher Evaluation Support System (T-TESS). This comprehensive, 
state-level framework is comprised of three main components: goal-setting and 
professional development, evaluation, and student growth measurement. 

In terms of goal setting and professional development, and evaluation, the T- 
TESS provides rubrics and resources for schools and teachers, including video 
training modules, handbooks, and conference templates. The third component, 
student growth measurement, requires districts to develop an appropriate system 
for determining student achievement growth, and this can be in the form of 
portfolios, student learning objects (SLOs), or value-added measurements (VAM). 
VAM, the most controversial approach — but arguably the most statistically 
sophisticated (Berliner, 2018) — has been used widely across Texas (and the US 
more broadly) for decades. It is designed to measure student growth from one 
year to the next, on state-wide standardized achievement tests, and link that 
growth to their teacher. In effect, student performance gains become a proxy for 
teacher effectiveness. The teacher’s VAM score is then used to inform evaluation 
and various personnel decisions, such as retention, professional development, and 
promotion. In the T-TESS, the overall student growth score is combined with 
observation scores to calculate the teacher’s cumulative evaluation score (which is 
perceived to be a measure of their effectiveness). 

While high stakes teacher evaluation is by no means new in Texas, the T- 
TESS is unique in three main ways: its use has been mandated for the entire state; 
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it was designed almost entirely by private agencies (which will be detailed in the 
analysis section of this chapter); and most of the T-TESS operates digitally via an 
online platform, with the T-TESS applications used for teacher observation and 
evaluation. In addition to features associated with teacher accountability, “T- 
TESS Cube” provides online professional development and training modules for 
teachers and appraisers to learn about the T-TESS and increase their respective T- 
TESS scores. Participation in these courses can also be tracked by school and 
system leaders. Altogether, the T-TESS generates and collates as many data points 
as possible to build a development and proficiency profile for each teacher, with 
each domain of a teacher’s professional practice (including their engagement with 
the T-TESS platform) able to be datafied and tracked. 

Our analysis of the T-TESS began with Selwyn’s (2015) prompt to engage with 


empirical work that strives to understand and account for the manner in 
which data are accumulated; to make visible the flow and circulation of data 
and begin to understand the ways in which data are then integrated back into 
everyday education practices. 


(p. 76) 


The current chapter picks up where Holloway and Lewis’ original analysis left off 
(see Holloway & Lewis, in press), by explicating the actors and agencies involved 
in the T-TESS, and exploring how the T-TESS might serve as a vehicle through 
which companies can prospectively engage in surveillance capitalism within 
public schooling systems. To this end, we collected and analyzed all publicly 
available resources associated with the T-TESS. This included, but was not lim- 
ited to, service contracts, company websites, terms of service agreements, imple- 
mentation guides, training materials, and reports associated with external partners 
(e.g., reports produced by SAS Institute Inc.). Guiding our analysis were three 
specific questions: (1) What agencies and actors are involved in the creation of the 
T-TESS?; (2) What types of data are collected via the T-TESS?; and (3) How 
might these data serve the processes and logics associated with surveillance 
capitalism? 


The T-TESS Ensemble 


The T-TESS comprises an ensemble of actors and agencies who work in part- 
nership with the TEA (the official state-level department for education in Texas), 
with these digital and analytic groups providing the technical expertise and ser- 
vices for calculating, analyzing, and evaluating teacher practice and performance. 
For this chapter, we have focused on three of the main private organizations that 
have contributed to the T-TESS, even though others have been peripherally 
involved over time: (1) SAS Analytics Inc., (2) Responsive Learning, and (3) the 
National Institute for Excellence in Teaching (NIET). Next, we provide a brief 
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description of these groups, including their primary services and involvement in 
the Texas education system. 


Rendering Teacher Performance as Data: SAS Analytics Inc. 


SAS Analytics Inc. is an analytic software company that owns the Education 
Value-Added Assessment System (EVAAS), which is a sophisticated value-added 
model (VAM). VAMs are statistical algorithms that operate by predicting how 
students should score on standardized achievement tests, comparing their actual 
achievement score to the predicted score, and attributing this difference (positive 
or negative) to the student’s teacher. This number represents the degree of 
“value” the teacher has added to (or subtracted from) student growth. 

The school year following the analysis, teachers receive a value-added report 
and a diagnostic report, which display statistics related to their growth measure, 
standard error, and effectiveness levels, as well as their students’ overall progress. 
These reports are expected to be used to inform teacher practice, even though 
the reports are received well after the students (and often teachers) in question 
have progressed to another grade level. All of these predictions are calculated 
from large data sets that allow data analysts to group students according to “like” 
testing histories. Using these historical trends, they then employ a predictive 
algorithm to determine how much growth a student should make in a single 
school year. This approach to measuring student performance putatively provides 
a more nuanced and unbiased view of a teacher’s influence on student achieve- 
ment, even though research has shown consistently that VAMs are still biased and 
unreliable (see Johnson, 2015). 

As of 2016, it was intended for SAS Analytics Inc. to partner with the entire 
state of Texas to provide VAM services. However, after the No Child Left 
Behind Act was reauthorized, and subsequently changed to the federal Every 
Student Succeeds Act (ESSA), states could return control of determining the 
student growth component of teacher evaluation back to local education autho- 
rities. Although Texas still required districts to maintain some form of a student 
growth model in their teacher evaluation systems, districts were permitted under 
ESSA to choose their own VAM. At the time of writing, SAS Analytics Inc. still 
partners with several districts in Texas to provide VAM services via SAS EVAAS. 
They have also partnered with schools and/or districts across the US, making 
them one of the most prominent for-profit providers of VAM methodology in 
the country (Amrein-Beardsley, 2008). 


Rendering Teacher Behavior as Data: Responsive Learning 


Responsive Learning is a for-profit, online learning platform that works with 
schools and other education agencies to develop online courses, such as profes- 
sional development courses for teachers. According to their website, the company 
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“brings together thought-leading authors, trainers, and practitioners to create 
engaging online professional learning for teachers that builds a bridge to instruc- 
tional practice in the classroom” (Responsive Learning, 2019, para. 1). TEA 
partnered with Responsive Learning to develop T-TESS Cube, an online pro- 
fessional development site where teachers can take various courses to help 
improve their performance as measured by the T-TESS. The online library con- 
sists of hundreds of videos, with many tagged specifically for the T-TESS pre- 
paration and development. A small number of these videos are specifically 
designed for the T-TESS itself (such as “Unpacking the T-TESS Rubric”), while 
the vast majority of the videos cover related topics that have subsequently been 
identified by Responsive Learning as being relevant to specific elements of the T- 
TESS framework. Principals or superintendents can request a free license that 
provides their teachers with access to a series of 15-minute mini courses that each 
align with the T-TESS rubric dimensions. All other videos and/or courses require 
payment for each participant, be they teachers, principals, or superintendents. 


Rendering Teacher Experience as Data: NIET 


The TEA has worked with the NIET for many years. The NIET is a non-profit 
organization that develops education programs closely aligned with various federal 
grant schemes. For instance, their TAP System for Teacher and Student Advance- 
ment (hereon referred to as the TAP System) has been particularly prominent within 
various federal grant schemes. The TAP System is a comprehensive evaluation 
system that incorporates ongoing observations, targeted professional development, 
VAM scores, and performance-based pay. TEA initially became involved with the 
NIET after winning a grant, after which TEA adopted the TAP System as their 
performance-based teacher evaluation system. At the conclusion of the grant, TEA 
kept major components of the TAP System rubrics and procedures, and these pro- 
vided the basis upon which the T-TESS was subsequently designed. More than 200 
schools in the US have since implemented the NIET’s TAP System, making it one 
of the most popular teacher evaluation system providers in the country. 

Together, SAS Analytics Inc., Responsive Learning, and the NIET have pro- 
duced the T-TESS framework, which defines what it means to be a public school 
teacher in Texas. Although each of these actors is located outside of the educa- 
tion profession, their analytic and digital experts have developed a technical pro- 
gram that defines quality for teachers and teaching in the state. Within the 
broader processes of datafication, these three organizations have been authorized 
by the state of Texas to develop and administer the T-TESS, which also entails 
the wholesale collection of private data when teachers engage with the platform. 
It is also worth noting that the obligatory nature of the T-TESS, insofar as core 
aspects of teachers’ work (e.g., goal-setting, evaluation, professional development) 
must be completed on the platform, means that T-TESS ensures a steady stream 
of user data for SAS Analytics Inc., Responsive Learning, and the NIET alike. 
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Discussion and Conclusion: The Possibilities for Surveillance 
Capitalism via the T-TESS 


Surveillance capitalism is, foremost, imbued with an economistic logic, which 
allows one to imagine the value of data not only for their present intended use 
(such as optimizing search terms on Google) but for what they might reveal in 
the future about the private interests, desires and actions of the human actors 
generating these data (e.g., voting intentions). It marks a significant departure 
from the modernist project in which one generates empirical data to respond to a 
particular known problem or question. Instead, data are cast as exchangeable and 
tradable commodities, to be bought and valued by different actors to provide 
solutions to as-yet unknown questions. Despite the behavioral predictions that 
might be derived from mining big data sets, or the actors or organizations who 
might seek to gain influence (or be influenced) by these predictions, it is the 
overwhelming collection and analysis of data that clearly animates this entire 
process. 

Turning to teacher accountability generally, and the case of T-TESS specifi- 
cally, we can see an entirely new and unprecedented means of datafying teachers 
throughout their career trajectory, a cradle-to-grave approach that encompasses 
initial teacher education, teacher evaluation, and ongoing professional develop- 
ment, with the resulting data collected, stored, and analyzed on a single software 
platform. In the context of teacher accountability, which is a central consideration 
in the US, the increased scope and scale of data collection via the T-TESS need 
not imply that all such data renderings of teachers are now considered “high 
stakes.” While the T-TESS clearly includes data used for purposes of teacher 
evaluation and promotion (or sanction), such as the value-added scores provided 
by SAS EVAAS, it can also collect and analyze data that is altogether more 
mundane, at least at first glance. For instance, the platform has the capability to deter- 
mine the time teachers spend on the learning resources provided via the T-TESS 
Cube, which resources they access, and how they interact with these resources 
(including time on screen, attempts made, and mouse clicks); that is, the secondary 
metadata that Zuboff (2019a) describes as “data exhaust.” 

While these data are unlikely to inform critical decision-making around a tea- 
cher’s nominal effectiveness, and thus impact their tenure or retention, these data 
are still valuable in the sense that they provide a digital record of a given teacher’s 
interactions with the T-TESS platform. Through the lens of surveillance capital- 
ism, it is these data that are potentially most valuable, at least in a monetary sense, 
as they can render teachers — their actions, their thoughts, their aspirations and 
fears — as data to inform subsequent targeted marketing of products and services 
(e.g., teacher professional development, university courses, professional member- 
ships, and mindfulness retreats). In some cases, it might not even be entirely clear 
what the data could be used for in the future. But, given that surveillance capit- 
alism positions data as a valued resource in itself, this may well distort the 
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incentive for private providers to (for instance) develop VAMs that are more or 
less useful for teachers, or even to improve VAMs to be more precise measures of 
teacher quality. 

Although the TEA may have primarily aimed for the T-TESS to provide 
feedback on teacher effectiveness as a notional benefit to public schooling, the 
data traces left by teacher activity on the platform still have the ability to be 
commodified and exploited for private profit by those who administer the plat- 
form. Moreover, this provides the developers of the T-TESS — all private for- 
profit and not-for-profit entities — with the ability to gather an unprecedented 
amount of data on the teachers involved, the danger being that the data are then 
used to better know and steer teacher behavior through strategic promotion and 
incentivization. We also note that financial motivations risk displacing, or at least 
pose a significant challenge to, more educative public objectives around enhan- 
cing student learning. 

To reiterate, our intentions here are not to impugn the motivations or inten- 
tions of the various for-profit (SAS Analytics Inc., Responsive Learning) and not- 
for-profit (NIET) actors associated with developing and administering the T- 
TESS ensemble. Nor do we seek to make specific claims to their actual present 
(or prospective) uses of the data, especially given that much of this information is 
obscured by commercial-in-confidence agreements and other contractual 
arrangements. Rather, we have sought to draw attention to the new and varied 
means by which all facets of teaching can now be rendered as data via the T- 
TESS platform, the prospective implications for how these data can be commo- 
dified and sold, and the potential ways this exposes teachers (as both individuals 
and private citizens) to targeted means of advertising and influence. As noted 
earlier in this chapter, while many of the tools and techniques of the T-TESS are 
similar to more traditional forms of teacher observation and evaluation, the 
comprehensive collection and storage of data on a single digital platform, devel- 
oped by multiple private organizations, creates its own unique set of concerns. 

Revisiting our earlier provocation about the influence of data analytics in the 
governing of social domains, and how this provided a previously unimaginable 
pathway for the election of President Trump, we would emphasize the danger 
associated with the ubiquitous nature of data analytics — and especially the com- 
modification of data — in our everyday lives. Using the T-TESS and its con- 
tributing partner organizations as a case in point, we have sought to demonstrate 
how datafication in schools provides fertile ground for surveillance capitalism to 
infiltrate and take hold, with these new private logics and practices often contra- 
dicting more publicly oriented motivations. As Zuboff (2019c) asserts, surveillance 
capitalism is on “a collision course with democracy” (para. 4). We must therefore 
acknowledge the privileged positions of the actors and agencies who have been 
authorized to participate in the Texas schooling system, as well as the financial 
and political opportunities that these privileged positions afford. As schooling 
becomes increasingly datafied, and thus becomes a more attractive target for 
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surveillance capitalism to (mis)appropriate, we must be ever vigilant against the 
possibility that data, rather than student learning, become the most valuable pro- 
duct of education. 


Notes 


1 The commodification of data is implicitly suggested by the popular and academic 
adoption of the verb “to mine” (that is, data mining), which we would argue equates 
data to any other such interchangeable or fungible commodity (such as oil, coal, or 
gold) that can be extracted, refined, and then sold for profit, with no concern given to 
the actual producer of the commodity. 

2 A more recent (and unfortunate) interpretation of the term “rendition” occurred in the 
context of the so-called War on Terror, in which the US Government-sponsored 
abduction and extrajudicial transfer of “illegal combatant” prisoners, often for purposes 
of detention and torture, was euphemistically described as “extraordinary rendition.” 
While of an altogether different nature, the rendition associated with surveillance 
capitalism is perhaps more dangerous because of the extent to which it has become 
normalized and accepted in contemporary society; that is, because of its sheer 
“ordinariness.” 
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GOVERNING BY DASHBOARD 


Reconfiguring Education Governance in the 
Global South 


Radhika Gorur and Ben Arnold 


Introduction 


There is something distinctly different being imagined by global aid agencies 
about how education is, or should be, governed in low-income nations. In the 
race to achieve the globally agreed goals under the United Nations Educational, 
Scientific and Cultural Organization (UNESCO)’s Agenda 2030, a discourse of 
crisis is being evoked. This crisis is not so much a learning crisis, as a data crisis 
(Gorur, 2019). The data crisis is epitomized in the words of two of the most 
influential global actors in education today: Silvia Montoya, Director of the 
UNESCO Institute for Statistics (UIS; which is the custodian of education data 
under the Sustainable Development Goals [SDGs]), and Luis Crouch, Senior 
Economist at RTI International. Montoya and Crouch (2018) assert that without 
the appropriate data on a range of indicators, the most well-meaning and deter- 
mined Ministers of Education in the Global South could not succeed in 
improving education. They identify that in many countries: 


... education ministers simply don’t have the data to avoid or even mitigate a 
global learning crisis, which is engulfing more than half of all children of 
primary and lower secondary school age, according to estimates by the 
UNESCO Institute for Statistics. 


(p. 2) 


Conjured up here is not just an abundance of data, but a particular arrangement 
of data in the form of a dashboard. The beleaguered education minister sits 
behind a dashboard of “navigation instruments” — lights flashing and machines 
beeping — urgently trying to correct a system headed for disaster. The visualized 
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data are ready to go — transformed into actionable levers that can be pulled and 
pushed to instantly elicit desired responses. 

Dashboards signal not just urgency, but agency. Like pilots in a cockpit, min- 
isters and policymakers are depicted as being able to control the destiny of the 
enterprise through deft management. This image of “governing by dashboard” 
(Bartlett & Tkacz, 2017) is not just an intensified version of the datafication and 
governing by numbers of the past three decades (Gorur, 2015; Lingard & Raw- 
olle, 2011; Miller & Rose, 2008); it represents a new phase, bringing together 
understandings of how governments ought to run with new technologies to 
enable decision-making. This mode of governance is not about periodic surveys, 
five-year plans, commissions, and reviews premised on a systematic, managerial, 
bureaucratic approach. Instead, governance is a pitched battle, fought in real time. 

This sense of urgency and agency, and the imaginaries of new modes of tech- 
enabled, data-rich governance, are now reflected in evolving global approaches to 
aid effectiveness. The failed African aid projects of the 20th century taught aid 
agencies that technical and financial assistance would be ineffective unless the 
political and policy environment and the bureaucratic machinery were also 
transformed (Kaufmann, 2009). Consequently, aid agencies sought to also reform 
these enabling conditions, namely the socio-political and administrative apparatus 
of recipient nations. This kind of “systems thinking” emerged as a paradigm shift 
in the field of development (OECD, 2019). Now, aid recipient nations are 
“partners” who “own” the development project. They are expected to reflect 
global priorities in policy formulation and to reform in-country systems to enable 
more effective deployment of aid (OECD, 2020). 

Even as recipient nations are delegated greater responsibility for aid effective- 
ness and urged to “own” the program, they are also closely monitored to ensure 
ongoing compliance. Recent decades have witnessed the rise of global data 
infrastructures that serve as sophisticated monitoring mechanisms (Jensen & 
Winthereik, 2013). Beyond the significant functions of transparency and 
accountability, these infrastructures provide common epistemic frameworks for 
aid and development. This enables the task of harmonization and alignment of 
goals. Data infrastructures fold in particular epistemologies of education govern- 
ance and nudge particular types of action. The systems thinking adopted by aid 
agencies has enabled data infrastructures to also serve as governance infra- 
structures. Dashboards are now increasingly popular features of global governance 
infrastructures, representing a style of governance that is vigorous, responsive, and 
results oriented. 

In this chapter, we explore the phenomenon of global governing by dash- 
board, with an empirical focus on the Global Education Policy Dashboard 
(GEPD) being developed by the World Bank in conjunction with DFID (the 
United Kingdom’s Department for International Development) and The Bill and 
Melinda Gates Foundation. The GEPD is set to be piloted in 13 countries across 
Africa, Asia, the Middle East, Latin America, and Eastern Europe (Root, 2019). 
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Currently five countries have been identified for the first phase of trials, with 
work at an advanced stage in Peru. The GEPD aims to facilitate the monitoring 
of the indicators that “drive” learning, from teacher preparedness and school 
management to “politics, [and] other actors that are affecting everything that is 
happening in the service delivery level” (World Bank, 2019d, p. 62). 

While the dashboard itself is not yet publicly available for viewing, a substantial 
amount of information is available in World Bank materials about the dashboard, 
as well as the indicators that underpin it. In our analysis, we have drawn on press 
releases, presentations and policy briefs about the GEPD; the World Develop- 
ment Report 2018 (the theoretical framework used for the dashboard); the 
Human Capital Index website (the parent project for the dashboard); and a range 
of other materials, including “Teach” reports and instructions, the Bureaucracy 
Lab website, the World Management Survey website, and the Systems Approach 
for Better Results (SABER) website. 

Williamson (2016) notes that dashboards are “the hybrid product of political 
aspirations to manage and orchestrate the flow of... data [and] the capacity of 
software to provide an apparently neutral, non-political interface” (p. 131). We 
build on this understanding to demonstrate that, beyond influencing governance 
through orchestrating data flows, dashboards are a sociotechnical imaginary that 
imagine and promote particular forms of governance. We seek to describe and 
highlight the politics of global data dashboards because their politics, though 
extremely consequential, are not always obvious or even visible. We have 
refrained, for the most part, from critiquing particular instruments because we 
wanted to maintain focus on demonstrating how the governance prescriptions 
embedded in such dashboards, which seek to improve aid effectiveness, might 
inadvertently be undermining some key principles of aid effectiveness. 


Dashboards in Governance 


Stephen Few (2007) states that “a dashboard is a visual display of the most 
important information needed to achieve one or more objectives; consolidated 
and arranged on a single screen so the information can be monitored at a glance” 
(p. 1). Originally designed for monitoring activity with a view to informing 
action, dashboards are also a means of dealing with information overload, since 
they present purposefully curated data on a single screen (Yigitbasioglu & Velcu, 
2010). Most widely deployed in business intelligence, dashboards are now used in 
a variety of contexts including health management, education, and smart city 
governance (Sarikaya et al., 2019). 

Muscular, tech-savvy, data-driven “governance by dashboard” is part of the 
contemporary sociotechnical imaginary of governance. Jasanoff and Kim (2015) 
describe sociotechnical imaginaries as “collectively held, institutionally stabilized, 
and publicly performed visions of desirable futures, animated by shared under- 
standing of forms of social life and social order attainable through, and supportive 


Governing by Dashboard 169 


of, advances in science and technology” (p. 4). The notion of sociotechnical 
imaginaries draws together the technical and the social as mutually productive. 
For Jasanoff and Kim (2015), sociotechnical imaginaries are not only about norms 
and discourses, but also about science and technology; consequently, the social 
and technical must be analyzed together. Understanding the relationship between 
the social and the technical requires also taking account of the “often invisible 
role of knowledges, expertise, technical practices and material objects in shaping, 
sustaining, subverting or transforming relations of authority” (Jasanoff, 2004, p. 4). 

The sociotechnical imaginary of “governance by dashboard” anticipates certain 
relations, effects and forms of authority. “They encourage more intensified forms 
of monitoring and analysis. Through their increased presence, they change the 
empirical basis from which decisions are made and also the criteria for what 
counts as a good decision” (Bartlett & Tkecz, 2017, p. 8). The Number 10 
Dashboard, designed for the Prime Minister of the United Kingdom, illustrates 
dashboard governance in action. “With a few taps or swipes of his fingers, [the 
Prime Minister] can see very quickly what important new information has come 
to light, how certain government services are performing and a selection of rele- 
vant and important news reports” (Cabinet Office, 2012, para. 4). 

Bartlett and Tkacz (2017) argue that “government, too, was becoming part of 
the big data revolution” (p. 7). The Number 10 Dashboard used “the latest 
technology to pull data from a huge variety of sources” (Cabinet Office, 2012, 
para. 8), and the Prime Minister “liked the statistical side, where we could give 
him quotable facts about what was going on. He found it useful to have a hard 
evidence base” (para. 6). The Number 10 dashboard gave the Prime Minister 
“instant access to GDP, bank lending, jobs and property prices plus polling data 
and Twitter feeds” (Gibbs, 2014, para. 6). 

Dashboards also usher in new opportunities for digital democracy and citizen 
participation. Many are open to the public, expressing government’s commitment 
to open communication and transparency. The visualization techniques used in 
the dashboard make data accessible to a wide range of users, and interactive 
interfaces allow citizens to provide suggestions and set visualization preferences. 
Perhaps the most visible and interesting use of dashboards has been made in 
governing “smart cities.” In the United Kingdom, for example, a stream of real- 
time data (including London Underground rail line status, the availability of bikes 
for hire, weather and air pollution levels, live local news, and stock market 
updates) combine on large displays — or iPad walls — to enable citizens to interact 
with the dashboard in passing. These dashboards also allowed the Mayor of 
London to “look over the capital digitally as well as physically” (Bartlett & Tkacz, 
2017, p.12). As Kitchin et al. (2015) report, “open data and dashboard initiatives 
are changing... the relationship between government and the public” (p. 7). 

In international development, dashboards have emerged as a popular tool to 
monitor outcomes in a range of fields, including health, education, the environ- 
ment, peace, justice, and security. The SDG Dashboard tracks the progress of 
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nations towards all 17 SDGs. The World Bank, The United Nations Children’s 
Fund (UNICEF), and DFID supported the development of several national and 
sub-national data dashboards with funding and expertise to ensure policymakers, 
bureaucrats, and other stakeholders have relevant, up-to-date information to 
enable good decision-making. The World Bank (2019a) believes dashboards serve 
as a form of accountability, providing donor agencies and local citizens easy-to- 
interpret data on the progress made towards development objectives. While 
development agencies have long been producing and publishing education data, 
these dashboards broaden and deepen the scope of data to encompass many dif- 
ferent aspects and levels of systems (often in granular detail), while representing 
this complexity in a simplified and user-friendly format. Consequently, dash- 
boards change “the empirical basis from which decisions are made and also the 
criteria for what counts as a good decision” (Bartlett & Tkacz, 2017, p. 8). 

Despite the growing popularity of governance by dashboard, a number of 
problems have also been identified. Sarikaya et al. (2019) argue that dashboards 
“bring together challenges of at-a-glance reading, coordinated views, tracking 
data and both private and shared awareness” (p. 1). There are epistemological and 
organizational consequences to governing by dashboards, as data scientists and 
visualizers introduce new vocabulary, and new ways of understanding, manip- 
ulating, and communicating with the data (Bartlett & Tkacz, 2017). Most 
importantly, “by introducing a new emphasis on metrics, indicators and measures, 
[dashboards] can create a greater focus on operational issues rather than longer- 
term strategic ones” (Bartlett & Tkacz, 2017, p. 5). 


The Sociotechnical Imaginary of the GEPD 


Launched in 2019, the GEPD seeks to provide low- and middle-income governments 
with the tools necessary to “better monitor the quality of their education systems” and 
enable “policymakers to take real-time decisions to ensure that all children are learn- 
ing” (World Bank, 2019e, para. 1). The project has four overarching aims: 


1. “develop tools to collect data at all levels of the system on a regular basis; 

2. embed innovative measurement approaches in a coherent, system-wide 
framework; 

3. present a user-friendly dashboard interface; and 


4. make resources available for implementation in all developing countries” 
(World Bank, 2019a, p. 2). 


To support policymakers in monitoring the education system via the dash- 
board, the GEPD project draws on particular, inter-related imaginaries of the 
education system, the classroom and the “user.” These imaginaries are inscribed 
into and operationalized by the indicators, tools and instruments used in the 
dashboard. In this section, we examine these imaginaries. 
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The Imagined “System” 


Dashboards inscribe within them a range of understandings about the factors that 
are relevant to education policy decisions and the relationships between these 
factors. The GEPD is underpinned by the World Bank’s change theory, which 
holds that: 


the learning crisis has multiple causes: poor service delivery in schools and 
communities, unhealthy politics and low bureaucratic capacity, and policies 
that are not aligned toward learning for all. To tackle the crisis and improve 
student learning for all, countries need to know where they stand on these 
three key dimensions — service delivery, policies, and politics. 

(World Bank, 2019b, para. 3) 


This new, broad approach radically links “the learning crisis” and school failures 
with service delivery policies, expanding the scope of reform objects. In this for- 
mulation, the key drivers of improved student outcomes are not better resourced 
classrooms, better educated teachers, child-friendly classrooms, or school nutrition 
programs, which have been the staples of previous reform. Instead, this new 
imaginary seeks to enhance student learning outcomes by focusing reform efforts 
on the whole system of education (Filmer et al., 2018). 

This “systems” approach looks beyond the typical unit of analysis (schools) and 
takes into account the various elements (politicians, bureaucrats, managers, policies, 
institutions, accountability systems, bureaucratic structures, financing, corruption, 
etc.) that constitute the larger structure of the “education system” (see Figure 10.1). 
The learning problem is translated into one of misalignment between different ele- 
ments of a complex system (Filmer et al., 2018). In this formulation, a range of dif- 
ferent technical and political barriers, including the complexity of the system, the 
large number of actors, competing interests, and the difficulty of moving out of low- 
quality equilibrium, all frustrate the development of the education system. On the 
other hand, if systems can overcome these issues and align different actors toward 
learning, remarkable learning outcomes can be achieved (Filmer et al., 2018). 

Emphasizing the importance of the interdependencies between diverse and 
multi-level factors on learning, the World Bank argues that “no single instrument 
pulls together data on all these areas” (World Bank, 2019a, p. 1; see Figure 10.2). 
The GEPD aims to remedy this lacuna by creating an instrument to “give gov- 
ernments a way to set priorities and track progress as they work to close those 
gaps” (World Bank, 2019b, p. 2). 

The GEPD focuses on four aspects of student learning: (1) learner preparation 
(a complex indicator that includes such things as access to pre-school education, 
poverty, and health); (2) teaching; (3) school management; and (4) infrastructure. 
The school-level aspects are combined with other system measures, as “the poli- 
cies and politics that determine the quality of service delivery,” ultimately 
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Politics 


Policies 


FIGURE 10.1 The GEPD’s conceptualization of the education system (World Bank, 
2019a, p. 1) 


contribute to the development of “a set of indicators that is comprehensive but 
also focused so stakeholders can pay attention to what is really most important” 
(World Bank, 2019b, p. 2). 

To realize this multi-level framework, GEPD draws on an eclectic combina- 
tion of global indicators and scales, including SABER, the Service Delivery 
Indicators (SDI), the Measuring Early Learning and Quality Outcomes 
(MELQO), the World Management Survey, and the Bureaucracy Lab surveys 
(World Bank, 2019b). In addition, classroom observations and interviews with 
various actors add finer detail. In this way, the dashboard aims to comprehen- 
sively cover the education system “from the level of the learner to the level of the 
system” (World Bank, 2019a, p. 2). 

While some of the data are gathered specifically for the purpose, many of the 
instruments employed in this dashboard have been developed at different times 
and for different purposes. For example, the World Management Survey was 
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FIGURE 10.2 The World Development Report: “technical and political factors” that 
“divert schools, teachers, and families from a focus on learning” (Filmer 
et al., 2018, p. 13) 


initially developed to measure management practices in manufacturing and then 
adapted for use in education. As such, these dashboards are “epistemological and 
methodological pastiche(s)” (Mattern, 2015, para. 31) that represent one of the 
many ways that the education system could be defined and understood. This 
pastiche is a result of the World Bank’s preference for research and instruments 
from the field of economics (most of those involved in the GEPD project are 
developmental or labor economists), the dashboard imperative of presenting all 
the key data on a single screen, and the trust in standardized scales and measures, 
some of which have already been in operation for several years. Thus, the GEPD 
operationalizes a different imaginary of governance, one that addresses learning 
from factors at the individual and school level, all the way to the bureaucracy, 
policy, and the political ethos. The ability of dashboards to present diverse 
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information from a variety of sources on one screen creates new relationships 
between data. Underpinning the numbers on the screen are epistemologies and 
world views that come to be inscribed into the data. Once established, these 
representations potentially become the “defacto... epistemology” (Kitchin et al., 
2015, p. 7) through which policymakers measure their systems, act upon them, 
and review performance. 


The Imagined Classroom 


Despite the widening of the reform focus to include the political, policy, and bureau- 
cratic functioning to address the learning crisis, the classroom — and more specifically, 
the teacher — remain key sites of reform. To measure the quality of the classroom, the 
GEPD uses an instrument called “Teach,” which is designed to capture what occurs 
inside classrooms (World Bank, 2019c). “Teach” makes a distinction between the 
structures of teaching — such as teachers’ qualifications and professional development, 
salaries, and years of experience — and “process quality,” which refers to student-teacher 
interactions. While the former variables have been part of input data in many surveys, it 
is the latter, “Teach” documents state, that are of more consequence. They cite a range 
of studies that showcase the difference made by particular teaching practices in middle- 
and low-income nations, and link specific teaching practices with an improvement in 
learning outcomes, citing effects of between 0.07 and 0.18 standard deviations (Molina 
et al., 2018, p. 3). Although it is recognized that improved student outcomes are 
dependent on a complex set of factors, in this formulation, differences made by teaching 
practices can be identified in granular detail. 

The “quantity and quality of instruction” is broken down into “(i) the time 
teachers spend on learning and the extent to which students are on task, and (ii) 
the quality of teaching practices that help develop students’ socioemotional and 
cognitive skills” (Molina et al., 2019, p. 2). The “Teach: Observer Manual” 
(Molina et al., 2019) explains how this is done: “Time on Task component, 3 
snapshots of 1—10 seconds are used to record both the teacher’s actions and the 
number of students who are on task throughout the observation.” This is com- 
plemented by the Quality of Teaching Practices component: “organized into 3 
primary areas...Classroom Culture, Instruction, and Socioemotional Skills. These 
areas have 9 corresponding elements that point to 28 behaviors.” These behaviors 
are the classified: “low, medium, or high, based on the evidence collected during 
the observation.” These are then represented on a “5-point scale that quantifies 
teaching practices as captured in a series of two, 15-minute lesson observations” 
(p. 2). To ensure that the observations are performed in a standardized manner, 
an observation tool was developed. Figure 10.3 is the aspect of the observation 
tool for recording the practices related to socioemotional skills. 

The World Bank’s imaginary elaborates observable behaviors that are expected 
to occur in similar ways in classrooms across the middle- and low-income world. 
Elements such as providing students with choice, encouraging goal setting, and 
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7. AUTONOMY Up | s} EE 
7.1 |The teacher provides students with choices L M H 
7.2 |The teacher provides students with opportunities to take on roles in the classroom L M H 
7.3 |The students volunteer to participate in the classroom L M H 
8. PERSEVERANCE Pie |e | a |b 
8.1 |The teacher acknowledges students’ efforts L M H 
8.2 |The teacher has a positive attitude towards student’ chalenges | L M H 
8.3 |The teacher encourages goal setting L M H 
9. SOCIAL & COLLABORATIVE SKILLS 1 2 3 4 5 
9.1 |The teacher promotes students’ collaboration through peer interaction L M H 
9.2 |The teacher promotes students’ interpersonal skills L M H 
9.3 |Students collaborate with one another through peer interaction | L M H 


FIGURE 10.3 Teach observation tool (Molina et al., 2019, p. 14) 


promoting peer interaction are seen as universally desirable behaviors that are as 
relevant in Pakistan as Rwanda. 

Despite the brevity of observations, “Teach” is seen to have multiple uses beyond 
measuring teacher practices, including “as a system diagnostic and monitoring tool” 
that can help “governments identify bottlenecks in service delivery, monitor the 
effectiveness of their policies, and focus efforts to improve teacher practices” (Molina 
et al., 2018, p. 3). As a professional development tool, “Teach” is able to “identify 
individual teachers’ strengths and weaknesses and coach teachers to improve their 
practice” (p. 3). Thus, it not only measures what occurs in classrooms, it is also an 
epistemic tool that is used to coach teachers to teach in ways that are valued by the 
instrument. When this occurs, learning is one-way, as observers learn nothing about 
other teaching methods that may be novel and culturally appropriate, nor do they 
learn whether the recommended strategies are problematic in specific contexts. 
Despite the acknowledgement of the importance of context in development work, 
such instruments are remarkably acontextual. 

“Teach” is one of several instruments used in the GEPD. Like “Teach,” each 
of these instruments brings its own set of assumptions and prescriptions, epis- 
temologies and ontologies, and they each encourage particular understandings 
and actions, irrespective of the contexts in which they are employed. 


The Imagined User 


One of the ambitions of the GEPD is to provide national policymakers with 
information about their education systems. Within the GEPD, the policymaker 
appears to be remarkably like the minister of education evoked by Montoya and 
Crouch (cited earlier). The World Bank (2019a) observes that education policy- 
makers in low- and middle-income nations are committed to enhancing student 
learning “often find themselves flying blind” (p. 1). 


They see the budget that goes into education and (sometimes) the learning 
that students come out with, but they lack information on the crucial factors 
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in between—the practices, policies, and politics—that drive those learning 
outcomes. The new Global Education Policy Dashboard initiative will fill 
that gap. 

(p. 1) 


Here, the unspoken assumption is that policymakers in low- and middle- 
income nations are not aware of the complexity of their education systems, nor 
are they able to deal with complex data. This is evident in a number of state- 
ments, such as the assertion that policymakers do not have the necessary evidence, 
and that they need to be assisted by external experts (Haus, 2019). Dashboards are 
designed on the basis of the perceived needs of inexperienced users, since they 
can succinctly and simply convey key information, through clever visualization 
techniques, while delegating complexity to the dashboard itself, black boxed and 
layered beneath the surface. 

Moreover, these policymakers are thought to operate in environments char- 
acterized by “poor service delivery... unhealthy politics and low bureaucratic 
capacity, and policies that are not aligned towards learning for all” (World Bank, 
2019a, para. 3). In contrast to high performing systems where “bureaucrats and 
teachers can devote much of their energy to improving outcomes for students” 
(Filmer et al., 2018, p. 15), policy actors in aid recipient countries are thought to 
have conflicting motives and interests. Filmer et al. argue that, “many systems are 
stuck in low learning traps, characterized by low accountability and high 
inequality” (p. 15). They continue that these low learning traps “bind together 
key stakeholders through informal contracts that prioritize other goals such as civil 
service employment, corporate profits, or reelection, perpetuating the low- 
accountability equilibrium” (p. 15). Such actors, Filmer et al. suggest, “juggle 
multiple objectives” and also rely on “each other in an environment of uncer- 
tainty, low social trust, and risk aversion” so that it is often in their interest “to 
maintain the status quo — even if society, and many of these actors, would be 
better off if they could shift to a higher-quality equilibrium” (p. 15). In this for- 
mulation, the policy actors and administrators who use the dashboards to drive 
change are themselves also targets of change. By making their own performance 
appear on the GEPD dashboard, these actors open themselves up to public scru- 
tiny as well as the scrutiny of international aid agencies. 

These depictions highlight the tension between the “partnership” and “own- 
ership” of aid effectiveness, and the desire to intervene and develop policymakers 
who are seen to be crucial to a project’s success. While, on the one hand, the 
GEPD aims to assist high-level national policymakers to take control of their 
systems, hold actors at each level to account and address misalignment, it is pre- 
mised on an assumption that the policymaker is incompetent, unreliable, and self- 
interested. The GEPD, then, becomes an example of “choice architectures” that 
encourage policymakers to make particular decisions that are perceived to be 
more appropriate (Jones et al., 2013). 
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Conclusion and Discussion: Contradictions and Conundrums 


The sociotechnical imaginary of education and policymaking mobilized by the 
GEPD is complex and based on a systems logic, which promotes a “realist epis- 
temology” in which “the world can be imagined as visualized facts” (Kitchin 
et al., 2015, p. 7). This epistemology ignores the performativity and the ontolo- 
gical politics of such representations (Pickering, 2017). Like all frameworks, the 
GEPD is based on specific, normative conceptions about education systems and 
how they ought to be governed. The GEPD’s ideal education system is fully 
aligned, each part working in harmony with the others. Although the tools and 
instruments used in the dashboard appear neutral and apolitical, decisions about 
which instruments to use, which items to select, and how to bring them together, 
are political decisions that have consequences for how the system is imagined and 
governed. 

The GEPD grapples with a series of contradictions and conundrums. The 
expanded focus on a range of system-level factors is premised on the idea that 
context matters, and that a range of contextual factors influence the effective 
deployment of aid. However, the GEPD deploys standardized instruments and 
benchmarks, which rely on standardized understandings of ideal classrooms, sys- 
tems, and policy processes. The use of standardized instruments is important for 
the World Bank and other international partners because they offer a way to 
monitor a nations’ progress in a comparative way, and they allow progress (or 
lack thereof) to be rewarded (or sanctioned). Using existing standardized tools 
also keeps costs down while expanding the scope of analysis, since these tools 
circulate in several other global indicator systems. When standardized tools are 
used to measure the phenomenon of context, it is inevitably configured by the 
standard, and important specificities are sacrificed. In a dashboard, these simplifi- 
cations accumulate and multiply. 

The GEPD’s contradictory assumptions about the policymaker represent a key 
conundrum of trust. While aid agencies must assume that the systems and officials 
to whom they are providing aid are trustworthy and knowledgeable, the GEPD 
project is premised on an assumption of their lack of knowledge and expertise, 
and their suspect ethics. Critiquing the GEPD’s assumption that policymakers in 
low-income countries are “flying blind,” Martin Haus (2019) insists that: 


the underlying disrespect for governments and bureaucrats of the targeted 
countries is astonishing... Policymakers in low- and middle-income coun- 
tries are not stupid. They often know at least as much, usually more than, 
World Bank consultants about their education system and its inner workings. 


(para. 8) 


The tacit view of the low-income nation policymaker as unsophisticated and 
requiring outside expertise is materialized in dashboards that present data in 
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digested and simplistic formats. A key feature of dashboards is that they use 
visualization techniques to present complex ideas in widely accessible ways. While 
this enables actors with varied levels of data literacy to access the dashboard, it also 
comes at a cost. The translation from the phenomenon to the “visualized fact” 
involves a series of complex, often opaque maneuvers, which has the potential to 
change the meaning of the data in subtle but consequential ways. When com- 
plexity is outsourced to different groups of experts such as statisticians and/or data 
visualizers, and then mediated through dashboard design, the policymaker is pre- 
sented with a limited range of possibilities for action. Data that are already sim- 
plified through the use of standardized tools are further translated and simplified 
through various representational requirements of the software, the size of the 
screen, the technological architecture, the adaptations required for Internet 
speeds, and the assumed data literacy of the user. As a result, the meaning of the 
data could become significantly distorted. The policymaker previously character- 
ized as “flying blind” might be worse off now, armed with data that look precise 
and complex, but whose richness and complexity are not guaranteed. 
Dashboards can, therefore, deeply impact the way problems are understood, 
because they are presented to the policymaker in curated, at-a-glance visual forms 
that make both the problem and the solution appear to be self-evident. The 
GEPD promises to present policymakers with distinctly practical knowledge, 
including the policy options that need to be addressed and the appropriate levers 
for action. In this imaginary, the ultimate authority to determine policy action is 
centralized, while the agenda-setting power is dispersed into networks constituted 
by the World Bank, its partners, and global technical experts and analysts. In this 
way, dashboards like the GEPD plug policymakers into global knowledge com- 
munities that claim to have the expertise to analyze, diagnose and improve every 
nation’s education system. However, this approach provides only a narrow fra- 
mework for policymakers to analyze policy choices, it minimizes the role and 
expertise of sub-national policymakers, and ignores the way that policies are 
constructed, namely through complex political deliberation and compromise. 
Metaphors matter. Conceptualizing policymakers as air traffic controllers or 
pilots evokes an image of governance as the work of powerful and passionate 
individuals, belying the apparent commitment to the notion of systems and 
complexity. Dashboard governance evokes a policymaking process that can be 
sped up, a sense that policy measures involve instant responses using real-time 
data and real-time action to achieve immediate results. However, quick fix solu- 
tions in education do not exist and the results of any decisions or changes in 
practice may take years to manifest. Prioritizing instant results could lead to a 
focus on the more superficial elements, and to seeking short-term gains at the 
expense of deep and sustained reforms that may take time to bear fruit. More- 
over, such solutions can only work if democratic political processes are bypassed. 
While the GEPD attempts to account for the complexity of education systems by 
gathering data about various levels of the education system, it presents a simplified 
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caricature of the processes through which policies in education are identified, 
developed, and approved. 

The global aid industry has become impatient and anxious that it may not 
reach the SDG4 goals by 2030. This sense of impatience spills over into the 
creation of dashboards that can speed up capacity building by delegating these 
capacities to the instruments and technologies of the dashboard. While the 
expanded notion of education governance as encompassing a complex set of 
factors is to be welcomed, GEPD’s instruments smuggle particular epistemologies 
and ontologies that may or may not be ideal for low-income nations. Unless 
utmost care is taken, the “politics by stealth” (Gorur & Addey, in press) of 
international governance dashboards has the potential to be yet another manifes- 
tation of neo-colonial policy prescription from the Global North. 
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NEXT GENERATION ONLINE 
ASSESSMENTS, TECHNICAL 
DEMOCRACY, AND RESPONDING 
TO DIGITAL DISRUPTION 


Greg Thompson 


Introduction 


Education systems, institutions, and individuals who work in them are witnessing 
a shift in the ways that intractable problems are framed, policy solutions offered, 
and how those policy solutions “hit the ground,” or are enacted in specific school 
and systemic contexts. As market logics promoting choice, competition, and 
accountability through transparency agendas have become dominant in the gov- 
ernance of education, there has been an increased emphasis on policy designed to 
reform education in specific ways. The suite of policies that aims to reform 
schooling has, more recently, become largely fixated on assessment as a means to 
both deliver improvements through altering behaviors and structures. Tradition- 
ally, these policies have focused on standardized testing data and test-based 
accountabilities as the key lever or “catalyst” for change (Lingard & Sellar, 2013). 
Recently, however, the centrality of the traditional standardized test has been 
challenged by the emergence of new generation, online assessments that utilize 
advances in digital infrastructures, algorithms, and adaptive technologies that aim 
to produce 21st century learners amid anxiety regarding skills required for the 
“jobs for the future.” Digital technologies are now a central vector of education 
reform, working across multiple discursive traditions as they promise greater 
individualization or personalization through their capacity to be responsive to 
individual student needs, while also improving the efficiency and effectiveness of 
instruction through such slogans as “just in time learning,” personalized learning, 
and adaptive assessment. 

In Australia, this can be seen in the ways that the National Assessment Pro- 
gramme — Literacy and Numeracy (NAPLAN) census standardized test shifted to 
become an online adaptive test (Masters, 2017; Thompson, 2017). Despite the 
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political enthusiasm for the online testing and positive reports on student moti- 
vation and subjective test experience in the online mode (Martin & Lazendic, 
2018a), significant concerns were raised regarding the mode equivalence of the 
tests; the NAPLAN technical report showed that there were significant differ- 
ences in the results depending upon whether students did the test online or in the 
traditional pencil and paper version (ACARA, 2019). These problems with 
NAPLAN Online appear to have energized debates around ending census stan- 
dardized testing, and have resulted in political leaders calling for new, more 
“intelligent” forms of assessment. 

One result of this has been the emergence of the Online Formative Assessment 
Initiative (OFAI) that is being developed in Australia by governments, statutory 
authorities, and private vendors as an “innovative assessment solution” integrating 
“resources, data collection and analytical tools in one ‘ecosystem’ that is easily 
accessible, interactive and scalable to meet future needs” (OFAI, n.d., para 3). 
Part of the justification for the OFAI is concerns regarding the consequences of 
NAPLAN itself, and the emergence of calls for new forms of adaptable, standar- 
dizable assessments that will signal the end of the traditional, standardized test. 

This chapter will proceed, first, by outlining the educational and policy con- 
texts in Australia in 2020. Second, an account of the implementation (and 
enactment) of NAPLAN will follow, before a discussion of the emerging dis- 
courses around digital technology as a policy solution for the “wicked problems” 
of schooling in Australia. Subsequently, a detailed analysis of the OFAI follows, 
with an emphasis on how it represents the problem of assessment and what 
solutions it proposes. Finally, two concepts will be developed, that of displace- 
ment and technical democracy, to argue that one critical consideration with new 
generation assessments like the OFAI concerns how susceptible it will be to the 
problem of the displacement of the teacher from the assessment process. The 
“black boxing” problem regarding new forms of adaptive, digital technology 
refers to the “knowledge problem” that occurs within systems “whose workings 
are mysterious; we can observe its inputs and outputs, but we cannot tell how 
one becomes the other” (Pasquale, 2015, p. 3). One solution is to adopt a more 
technical, democratic focus that locates the data users/decision-makers as essential 
stakeholders within these systems. This is what is meant by technical democracy 
in assessment systems (Callon et al., 2009; Thompson et al., 2019). 


Education Reform in Australia 


A common feature of many reform agendas is the leveraging of data to drive 
behavior. Datafication, which has changed in scale and intensity as faster and 
more extensive digital tools emerge, uses the data that schools produce as “catalyst 
data” to elicit specific responses in schools and systems (Lingard & Sellar, 2013). 
The most obvious example of this is the use of test-based accountabilities in sys- 
tems and schools. This catalyst data often have perverse effects, as schools and 
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systems respond to the datafication of value by strategizing how to get the max- 
imum returns from data, which often stand as a proxy for school and teaching 
quality. These perverse consequences can include teaching to the test, narrowing 
curriculum focus and choice, displacement of broader educational goals, and 
increased stress and anxiety experienced by students, teachers, and principals 
(Thompson, 2013). 

In Australia, the election of the Rudd Labor Government in 2007 crystallized a 
“national system of schooling” (Lingard, 2010, p. 131; see also Savage & 
O’Connor, 2019) that used data, test-based accountability and financial incentives 
(Klenowski & Wyatt-Smith, 2012) to effectively set up a national bureaucracy 
governing education. The values of these reforms were organized around human 
capital theory. With such theory, “the market is taken as the grand metaphor for 
all human activity, sees education’s value as stemming primarily from its capacity 
to generate capital that can be traded for jobs and wealth in the marketplace of 
the global economy” (Clarke, 2012, p. 181). Clarke goes on to argue that the 
politics of policy at work in Australia naturalizes “social logics of competition, 
atomization, and instrumentalization” and inevitably “glosses over the tensions 
between its aspirations for inclusiveness and its preference for audit and account- 
ability mechanisms” (p. 187). 

While the Australian Constitution guarantees that the States manage schooling 
in their jurisdictions as a residual power, at least since the 1970s, there has been a 
creeping nationalization as the Commonwealth has assumed more and more 
control over schooling. For example, post the 2007 election the new Federal 
government set up three powerful organizations that texture this nationalization. 
First was the Australian Curriculum, Assessment and Reporting Authority 
(ACARA), a statutory authority responsible for the Australian Curriculum, 
national assessment including NAPLAN, and reporting on schooling in Australia. 
Second was the Australian Institute for Teaching and School Leadership (AITSL), 
a company wholly-owned by the Commonwealth of Australia responsible solely 
to the Minister for Education and responsible for initial teacher education 
accreditation, the cultural competency of teachers, the development of resources 
to support high quality professional learning, assessing and evaluating links 
between the Australian Professional Standards for Teachers and teacher effective- 
ness, and strengthening the preparation and ongoing support of school leaders and 
principals. Finally, in 2010, Education Services Australia (ESA) was formed, a 
national not-for-profit company owned by the state, territory, and Australian 
Government education ministers to provide technology-based services for edu- 
cation. These services extend to researching; testing and developing effective and 
innovative technologies and communication systems; promoting e-learning and 
supporting national infrastructure to ensure access to quality-assured systems and 
content; and interoperability of individuals, entities, and systems. 

These three organizations exemplify a defining feature of this national educa- 
tion system. It shows the centrality of data infrastructures to enable a variety of 
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practices. These include linking systems together through common data collec- 
tion and reporting standards, the use of common data to report on and compare 
systems and sub-systems, and the use of comparative data to track and “steer at a 
distance” actors within the various jurisdictions that constitute the Australian 
national schooling system (Heffernan, 2018). There are three noteworthy ele- 
ments of this infrastructure. First, a central aim is governance by comparison 
which necessarily requires the capture, storing, and analysis of data in order to 
make jurisdictions commensurable. The second, the structure of these governance 
entities is corporate rather than statutory. Finally, the emphasis on e-learning, 
digital solutions, and commercial innovation is best exemplified by ESA’s man- 
agement of the National Schools Interoperability Program (NSIP). NSIP was 
established in 2010 to support the interoperability of information systems used by 
government and non-government schools and school systems across Australia. 
NSIP uses interoperability standards to enable data sharing between schools and 
school systems (Lingard et al., 2017). 

This data infrastructure has created new markets and partnerships in Australian 
schooling for education technology companies and consultants (Sellar, 2017). The 
invisible work of data, or information ontologies, has to be a critical focus of 
research amid the growing implementation and joining up of data management 
systems across various scales, from individual schools to national school systems. 
Ontology has a dual sense in this context, first as a “philosophical concept 
describing the metaphysical questioning of being,” but also as it has been 
“developed in the field of information science where it has a distinct definition as 
a conceptual schema or simplified model of a world” (Sellar & Thompson, 2017, 
p. 494). Ontologies “enable the formalization of data... that can be commu- 


? 


nicated within and between systems,” while also laying claim to how actors or 
agents come to understand the nature and purpose of the systems that they 
operate within (p. 495). 

These data ontologies inform the production of standards to make the collec- 
tion of multiple forms of data sharable and communicable across multiple systems 
and contexts. Interoperability standards produce new possibilities for digital tech- 
nologies to be developed that can be used across jurisdictions, reducing the time 
and costs of product development and enabling generic solutions that eliminate 
specificities of context. Of interest is the ways that the NSIP enables policy to be 
constructed through partnerships between politicians/bureaucrats and commercial 
educational technology vendors. Vendors are able to use interoperable data to test 
their products, while systems are (hopefully) provided with digital products and 
services responsive to their needs (Lingard et al., 2017). 

The “data turn” in education policy represents a new mode of governance as 
comparison became a central element of it (Ball, 2006). This mode of governance 
is affected not just through testing regimes, but through the ways that these 
regimes combine with, and intensify the effects of other policy instruments. 
Consider, for example, the ways that the publication of results in league tables 
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turns standardized assessments into high-stakes assessments. The ways that assess- 
ment has become central to policy is worthy of note for two reasons. First, 
assessment is seen to be a device to drive behavior within systems, and as such is seen 
as a means to direct action. Second, if we were to typify the ways that policy con- 
ceptualizes assessment, it would be fair to say that most of the emphasis is on relia- 
bility (understood as the consistent and stable production of scores), rather than 
validity (understood as the inferences made from the data assembled). As will be 
detailed below, while there were many recognized problems with NAPLAN, these 
were not about standardization, but about inferencing and decision-making based on 
those inferences. It is noteworthy that the NSIP slogan on their website is, “The 
right information in the right place at the right time” (https://www.nsip.edu.au). 


Problems with NAPLAN Census Testing 


NAPLAN has been a feature of Australian schooling since 2008, with the 
MySchool website that published and compared school results first online in 
2010. The tests were aimed to “create a learning environment that encourages 
innovation and excellence from school leaders, teachers and students” (Rudd & 
Gillard, 2008, p. 31). The MySchool website was intended to provide students, 
parents, and teachers with evidence needed to make “informed choices” about 
learning programs, schools, and school enrollments (p. 31). 

The National Assessment Program’s aims are “to help drive improvements in 
student outcomes and provide increased accountability for the community” 
(ACARA, 2011, n.p.). The first aim of driving improvement is based on the logic 
that testing data can drive school improvement because it provides “detailed 
information about how [students] are performing, and they can identify strengths 
and weaknesses which may warrant further attention” (n.p.). Furthermore, for 
systems and bureaucracies, 


the NAP provides Ministers of Education with information about the success 
of their policies and resourcing in priority curriculum areas. The NAP also 
provides ministers with the capacity to monitor the success of policies aimed 
at improving the achievement of different student groups, such as Indigenous 
students. 

(ACARA, 2011, n.p.) 


The second claimed benefit is that the testing benefits the public because of its 
“accountability function” as “Australians can expect education resources to be 
allocated in ways that ensure that all students achieve worthwhile learning during 
their time at school” (ACARA, 2011, n.p.). Together, the intertwined logics of 
comparable data and teacher/school accountability are expected to “inform future 
policy development, resource allocation, curriculum planning and, where neces- 
sary, intervention programs” (n.p.). 
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Since its inception, there has been widespread criticism of NAPLAN as high 
stakes tests can have a negative impact on pedagogy, curriculum, and the stress 
levels of students and teachers (Cumming et al., 2016; Roberts et al., 2019). 
Researchers have argued that “teaching to the test” has become a part of teachers’ 
practice within the classroom and the focus of the curriculum has narrowed to 
the specific outcomes that are expected to be tested (see Lingard et al., 2016). 
There is concern that these factors may lead to a less diverse and creative class- 
room environment, and ultimately, a narrowing of the goals and purposes of 
education. Of particular concern is the increased pressure that teachers report to 
ensure students achieve certain metric outcomes as measured by the NAPLAN 
tests (Rogers et al., 2018; Thompson, 2013). The pressure felt by teachers exists 
in part because of the use of students’ NAPLAN results as surrogate evaluations of 
teaching performance (Hardy & Lewis, 2017). 

Although some have argued that the change to accountability structures has 
been beneficial, and perhaps even necessary for the teaching profession, others 
have suggested that test-based accountability only serves to “narrow” the curri- 
culum and bring the commencement of formal schoolwork to earlier years of 
schooling (Gable & Lingard, 2015; Roberts et al., 2019). This can result in stu- 
dents having to “grow up fast” as they are forced to cope with more rigid and 
formal learning. Significantly, other key aspects of a broad education may be 
neglected as teachers are required to give more attention to NAPLAN style 
questions (Lingard et al., 2016). Other authors have argued that this type of 
accountability can contribute to the perception of a “crisis” of teacher quality in 
Australia (Mockler, 2014, 2016). NAPLAN has had an impact on the way equity 
and similar social justice constructs are defined and acted upon as simply the 
strength of the correlation between students’ socio-economic backgrounds and 
test scores (Lingard et al., 2014). 

One of the most telling criticisms of NAPLAN has concerned its usefulness as a 
diagnostic tool. Teachers found that they received the results too late, sometimes 
six months after the students sat the tests. In the meantime, students had learned, 
developed, and made progress, such that it was only helpful in a very small 
number of cases (Lingard et al., 2016). This criticism partly led to ACARA 
moving towards online testing, which better enabled automated marking. 
NAPLAN Online was trialed in 2018, and in 2019, ACARA reported that 50% 
of schools sat NAPLAN Online. Interestingly, ACARA extols that “feedback 
about the online test has been positive, with students saying they find the format 
engaging” (ACARA, 2018, n.p.). 

While students have found the online version more engaging (Martin & Lan- 
zendic, 2018a) and reported higher levels of motivation in the online assessment 
(Martin & Lazendic, 2018b), there have been ongoing criticisms of the online 
test. These include a reported mode effect, which means that there can be dif- 
ferent results depending upon whether students sit the online or paper version 
(Thomas, 2019), and problems with computer hardware and/or connectivity 
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problems, the shutting out of many students, and the machine marking of written 
responses (Perelman, 2018). These concerns led many Education Ministers to 
express concern about NAPLAN online. For example, in Victoria, Education 
Minister Merlino removed Victoria from the online assessment, saying “despite 
states and territories raising repeated concerns about the test, the national bodies 
responsible for the administration of NAPLAN online have let us down for the 
past two years” (Carey, 2019, para 17). These and other criticisms have led some 
to argue that we are witnessing the end of census testing in some countries. In 
Australia, there is a move to augment testing by a new generation of online for- 
mative assessments that are fundamentally standardized, yet able, as claimed by 
supporters, to be responsive to context, learners, and classrooms. This com- 
plementary assessment is best represented by the work in progress on the OFAI. 


The Online Formative Assessment Initiative 


The core aspect of the OFAI is the use of learning progressions. These progres- 
sions were developed by ACARA and the New South Wales Department of 
Education based on the observable elements derived from the Australian Curri- 
culum to “efficiently and effectively identify where students are in their learning” 
and to assist teachers to “make informed decisions about what to do next” based 
on that information (ACARA, ESA, & AITSL, 2020, n.p.). The learning pro- 
gressions are expected to capture the skills, understandings and capabilities 
expected in the curriculum that develop as a result of engagement with a learning 
task. These skills, understandings, and capabilities are typically acquired as an 
aspect of mastery of a particular part of the curriculum. The OFAI Information 
Sheet produced by ACARA, ESA, and AITSL says that the learning progressions 
“are robust and evidence-based and have been validated using empirical data 
where available” (n.p). 

The aim of the OFAI is to produce a technical “ecosystem” where all of the 
information, resources, tools, and assessments that a teacher needs are available via 
one online portal. This technical ecosystem, 


will assist teachers to identify where students are, what they need, and what 
resources will be most useful to support learning progress. Teachers will have 
seamless online access to the data and the resources they need to adapt their 
teaching practice to meet the needs of their students. These digital tools can 
complement the ways teachers are already working. Students will also have 
access to information about their own learning progress. 

(ESA, 2020, n.p.) 


OFAI is organized around a series of formative assessment tasks. The information 
sheet outlines that the technical ecosystem consists of three components that 
work together. The first of these is a series of online tools such as “tests, quizzes 
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and other digital content” that use the learning progressions as “the common 
content base for the initiative.” The second, named as the integrated system, 
utilizes a “common measurement scale” so that all information collected via the 
online tools “can be compared and connected with the insights derived from 
other assessments that are integrated with the system” (ESA, 2020, n.p.). In other 
words, the common measurement scale creates a commensurable system that 
enables comparisons across the national system of schooling. The third compo- 
nent is the creation of a digital portal that collates all evidence from student 
learning and enables easy monitoring and plotting of student learning against the 
learning progressions. A key part of the OFAI is the development of professional 
learning activities and resources “for teachers and school leaders to support and 
assist effective implementation” (n.p.). 

The most interesting part of the assessment concerns decision-making. The 
ecosystem aims to create a “suggestion engine” that lists resources matched to 
individual student needs that teachers can give to students. Essentially the eco- 
system hopes to use machine learning, matched against the learning progressions, 
to deliver a personalized teaching intervention (targeted teaching) based on plot- 
ting students’ learning against both the learning progressions and against other 
students. 

While the technical documentation that accompanies the website uses Kane’s 
(2006) theorization of validity as an individual argument based on each inference 
made from an assessment (O’Leary, 2020), it is telling that the website itself makes 
the simplistic claim that its assessments are “reliable and valid,” seemingly to 
suggest that validity is a property of tests, rather than their inferences (ESA, 2020, 
n.p.). This remains an ongoing problem with many centralized approaches to 
assessment — how is it that validity is being understood, enacted, and given back 
to the profession to take ownership of? As I will argue at the end of the chapter, 
seeing validity as a both a technical and a democratic problem or possibility, 
suggests a way that many of the unintended consequences that have plagued 
standardized testing could be avoided or alleviated. 

The OFAI is a next generation assessment that aims to target teaching and 
learning through making the interventions more personalized, and therefore 
engaging. This is an example of the emerging vector of education reform named 
as the logistics of engagement that uses adaptive assessments and algorithmic 
feedback to keep students engaged and moving through the required syllabus. 


The Logistics of Engagement 


Many education systems remain paradoxically torn between the problems of 
“basic” skills such as literacy and numeracy and 21st century skills such as crea- 
tivity and computational proficiencies. It is not unusual to find policy papers 
extolling back to basics approaches sitting next to visionary talking about the jobs 
that do not as yet exist. At the same time, classrooms, schools, and school systems 
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are opening themselves up to external forces in ways perhaps unseen in their 
relatively short history. Data, evidence, informed decision-making, target setting, 
value-added measures, and school rankings are just some of the ways that tech- 
nology is reshaping school practice and producing new subjectivities. In 2014, the 
Education Technology Industry Network (ETIN) of the Software & Information 
Industry Association (SIIA) commissioned a report that looked at the education 
technology market in P—12 schools in the United States (Richards & Struminger, 
2014). The report estimated that in 2013, the industry was worth US$8.38 bil- 
lion. The largest category was in assessment, which was estimated to have earned 
US$2.5 billion in the period (Richards & Struminger, 2014). Within this, the 
report found that Personalized Learning Environments (PLE) that “included 
incorporating the use of blended learning environments, expanded use of digital 
learning platforms, and using adaptive instructional software” were one of the 
growth areas for profit in the sector in the coming years (Richards & Struminger, 
2014, p. 39). This emerging market included the growth of cloud-based services, 
app development for mobile devices, “big data” analytics, and social networks 
customized for educators. 

Learning personalization (LP) is one of the key goals of the next generation of 
digital assessments in education. The US National Education Technology Plan 
(US Department of Education, 2010, p. x) defined personalized learning as those 
“learning experiences...that mirror students’ daily lives and the reality of their 
futures.” Furthermore, technology is often assumed to enable a static, linear pro- 
gression of learning that “is paced to learning needs, tailored to learning pre- 
ferences, and tailored to the specific interests of different learners [and]... the 
learning objectives and content as well as the method and pace may all vary” (US 
Department of Education, 2010, p. 12). What seems missing in these accounts is 
recognition of the speed and scale of the computations possible, and the how 
these may be thought of as “opening up new ways of thinking and acting, new 
problematizations, new authorities, new technologies and new conceptions of the 
subject” (Rose, 2004, p. 273). Thinking that LP is simply a process that adds to 
the existing repertoires of educational institutions seems somewhat naive; radical 
transformations to core processes, relationships, and normative ideals are more 
likely. 

LP is characterized by the 


real-time rendering of learning resources and social suggestions based on the 
profile of a learner through the ability to track each learner’s physical and 
online interactions, analyze skills and competencies, and then compare lear- 
ner knowledge with the mapping of knowledge in a discipline. 

(Siemens, 2013, p. 1392) 


LP focuses on constructing the learner through datafied patterns that profile and 
can adapt to offer the learner’s next move or activity. LP is that process in which 


Next Generation Online Assessments 191 


educational resources and learning environments are continually modified, such 
that each learner feels motivated to attain the highest level of knowledge possible 
for that learner in a specific field of knowledge. In the OFAI, this is the ultimate 
aim of the technical ecosystem, the promise of improved, if not complete, learner 
motivation, as the learning content, environment, and tasks are continually 
updating and responsive to the learner through building profiles due to the 
datafied patterns emerging. 

Computerized technologies that enable the collection and analysis of large data 
sets in short periods of time result in a surface where the continual modification 
of activities and objectives is designed to maximize engagement. This constant 
deformation, to induce the greatest and most effective effort on the part of that 
learner, a “logistics of engagement,” is crucial to the individuating path that 
unfolds, while also enfolding a learner. If learning personalization is to involve 
continual modification of learning environments, then various means of making 
that environment personal (and effective) for the learner must be available in the 
system. The learning personalization system must contain a variety of possibilities 
with respect to: presenting a learning environment, modifying the features any 
learning agents that may be at work in that environment, altering the grammar of 
the written or spoken material involved, changing styles of presentation (verbal to 
visual), and so on. These modifications need not be done to overcome a lack of 
motivation, as it is also done to effect continuing motivation. This can involve 
intrinsic rewards and extrinsic rewards. Making a learning environment increas- 
ingly sophisticated provides intrinsic rewards, whereas badges and leveling-up are 
examples of extrinsic rewards. 

Understanding LP, firstly, requires “mapping” the content or knowledge field 
of what is being learnt against which the learning behaviors and preferences can 
be profiled. OFAI does this through the learning progressions, derived by 
ACARA and the New South Wales Department of Education from the Aus- 
tralian Curriculum. This datafication of the field of knowledge requires re-ren- 
dering its attributes to be captured through digital representations — the process 
that ACARA refers to as identifying the “observable elements” of progression 
through the Australian Curriculum. This can involve tagging by experts, but can 
also involve automated tagging processes, which either employ tags created by 
experts or mine the Internet to map the knowledge domain. This is referred to as 
knowledge tracing, which “involves representing the knowledge required to 
master a domain, and, from traces of online user behavior, diagnosing user 
knowledge states as a profile over those elements” (Pirolli & Kairam, 2013, 
p. 140). The progressions “were developed using evidence-based research in con- 
sultation with literacy and numeracy experts and practicing teachers’ (ACARA, 
ESA, & AITSL, 2020, n.p.). 

Finally, and of particular significance, knowledge tracing and learning behavior 
operate through and on learner motivation. Motivation, or learner disposition, is 
datafied and incorporated in such a way as to enable the continual modification 
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of a learning environment so that it unfolds, reportedly to most effectively engage 
that learner. Personalization is not simply a matter of overcoming declining levels 
of motivation, but also of inducing even higher levels of motivation from those 
who are already motivated. An effective learning personalization process is 
designed to induce the highest levels of motivation possible for all learners. It 
must have the capacity to adapt in whatever ways are necessary to maintain the 
highest possible level of motivation as often as possible. 

What remains interesting about the OFAI, and presents a potential new fron- 
tier in the relationships between teaching, learning, and assessment and the 
necessary underpinning evaluative expertise, concerns the way that it aims to 
promote, or nudge, teacher decision hyphenate making in light of teaching 
methods referred to as high-impact teaching practices. The centrality of cognitive 
and learning sciences in designing the algorithms' that work to profile students 
and then select appropriate resources for teachers remains somewhat of a “black 
box.” What is not clear is how decisions are made, how matching occurs, and 
how profiling of individual learners, teachers, and classroom contexts is oper- 
ationalized and subsequently modeled. While the OFAI might best be considered 
the most recent example of the use of technical and technological prostheses that 
aims to improve efficiency and effectiveness (and to an extent, save time for 
teaching by automating processes such as assessment, ongoing planning and 
modification for student learning) the effects of this remain to be seen. Ball (1994, 
2006) usefully reminds us that research should always be alert to the first and 
second order effects of policy shifts: 


First order effects are changes in practice or structure (which are evident in 
particular sites and across the system as a whole). And second order effects are 
the impact of these changes on patterns of social access and opportunity and 
social justice. 

(Ball, 2006, p. 51) 


With than in mind, I suggest three areas of possibility, and perhaps caution, 
regarding the effects of the OFAI. The first regards data and ownership, the 
second addresses the problem of the possible displacement of the teacher through 
seeing validity as a problem requiring a technical democracy approach, and finally 
reflecting on how it is that teaching and learning will change as a result. In a 
sense, I am mindful here of Guattari’s (2013) caution in regards to technology: 
the important question is not that of destructive change caused by exponential 
increases in technical-machinic power, but rather why, given dramatic technolo- 
gical development, we still see “a reinforcement of previous systems of alienation, 
an oppressive mass-mediatization, infantilizing consensual politics” (p. 4). The 
ongoing dilemma relates to expertise; decision-making and the capacity of data- 
users to make decisions, sharpened through the formative assessment ecosystem. 
Equally, how is it that students come to own their data, and be assisted to make 
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decisions about the purposes to which it is used and by whom. Technical 
democracy goes beyond labeling the problem of black boxing (hiding the work- 
ings of algorithms that shape technological decision-making) to suggest a number 
of solutions to digital disruption. 


New Possibilities: Owning Data and Responding to the 
Displacement of the Teacher 


The idea of first and second order effects sets up an ongoing provocation for 
thinking about the relationships between assessment, datafication, commerciali- 
zation, and the challenges that these present for the expertise of the teacher. This 
is the assessment problem of displacement and unintended consequences (Thomp- 
son et al., 2019). Displacement is the spatio-temporal arrangement of expertise 
and the “distance of travel” from the site of technical expertise to the site of 
pedagogical decision-making. One way to understand why there are unintended, 
and sometimes perverse, consequences when policy is put to work in specific 
contexts, is that the work is often done to decision-makers (such as teachers and 
principals), rather than with them. The political expedience of the slogan that 
more and better data will fix education fails to recognize this problem of dis- 
placement. In many fields there is an opposite effect where too much data (the 
data deluge) actually makes decision-making more difficult. Making better deci- 
sions with data is critical; paradoxically as displacement increases, we see the rise 
of unintended and perverse consequences. 

If we return to traditional standardized testing, it has become very clear that the 
evaluative expertise needed to understand, and critically use, the data produced 
remains problematic at the classroom level. When researching the effects of NAPLAN 
on school communities, I asked many teachers what they knew about testing itself 
(Thompson, 2013), particularly given their frequent concerns about NAPLAN, and 
how the publishing of school data for comparison, created environments where they 
felt their pedagogical choices were coerced with a diminished professional expertise. 
Always lurking within these testing and reporting systems is the problem of account- 
ability for results and how this influences behavior, often in unintended ways. This is 
what I would term the “commonsense” argument about datafication, namely that it 
strips teachers of their expert role (or professional judgement). 

However, I think there are problems with this argument. First, it oversimplifies 
a complex history of teacher expertise. Second, historically, teaching has had a 
relationship with data in one form or another (at least since schools became 
influenced by Taylorist notions of efficiency in the early days of the 20th century; 
Callahan, 1964). While we have always had data, we have never had so much, so 
available, and so malleable to calculation. Also, while expertise has always been 
conferred/deferred, the distance from the classroom (in both an experiential/ 
somatic sense and an epistemological sense), where expertise is wielded, has 
grown alarmingly. 
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This leads us to the opportunity of the OFAI. On one hand, expertise recog- 
nized in the OFAI remains held by a small group of test-designers, psychome- 
tricians, and bureaucrats whose work then authorizes certain modalities of 
teaching. The list of “consultants,” as outlined in the OFAI documentation, 
represents the usual suspects involved in the “metricization” of schooling (Mau, 
2019). There is something of a perfect storm here, the national system of 
schooling, and the standardization (the data ontology) of data enabled by NSIP is 
producing new opportunities for commercial access to schooling. As these tests 
become even more complex (for example, NAPLAN morphed to a computer 
adaptive test), the technical expertise required to understand, and critically eval- 
uate these tests, will be so advanced that it is even less likely that teachers will be 
able to muster the insight into assessment required of a profession. This leads to 
significant problems of expertise with no simple solutions. Of course, much cor- 
poratization operates through providing solutions to real and imagined crises. For 
example, in Australia, testing has acted as a Trojan horse for new, commercial 
expertise in testing situated in EdTech companies and other edu-businesses. 

On the other hand, what stands out in the OFAI is the careful and methodical 
way that schools, principals, and teachers have been consulted during the design 
of the assessments. The website outlines that a core design principle is “stake- 
holder consultation and engagement into all stages of the project” and the web- 
site lists schools that have come on board as partners (ESA, 2020, n.p.). Generally, 
stakeholder engagement in these matters is usually kept to the “front end” of the 
design, perhaps because the overarching emphasis is on assessment reliability 
rather than validating inferences. The opportunity for the OFAI is to consider 
how to include teacher expertise in decision-making, and indeed the ongoing 
validation of decision-making, through all stages of the assessment ecology’. 

As we have argued previously with regards to international large-scale assess- 
ments (ILSAs), the idea of ongoing and thorough validation studies conducted by 
teachers who understand best not the inferences that should be made, but the 
inferences that are being made, do more than simply identify the problem of the 
assessment black box (hidden judgement and decision-making), it opens up a new 
form of accountability to data. Thompson et al. (2017) note, “ILSAs can involve 
teachers and their various collective voices (such as professional associations, sub- 
ject associations and teacher unions) playing a role in testing and improving the 
validity of inferences in their contexts” (p. 62). They suggest further that the 
validity of the tests would be enhanced, if domain experts (teachers) and technical 
experts came together to learn from each other. It is not being suggested that 
teacher voices should be “privileged over technical experts,” but that the coming 
together of both would enhance test validity (p. 62). The danger, of course, is 
that this critical work is handed over to commercial entities, often outsourced by 
regulatory authorities such as ACARA and ESA, whose interest will always be 
modified by commercial realities such as profit, market share, and leveraging 
further financial opportunities. 
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If displacement might be meaningfully countered by a technical democracy 
approach to validation, another glaring issue concerns the ownership of data that 
is produced by (largely) students, and used by teachers for education purposes. 
The types of arrangements favored by ESA, and that are evident in both the 
NSIP and the OFAI, concern a particularly blurred public-private relationship. 
Vendors are central to both the design and technical governance of these systems. 
These corporate and business entities access public school data, including that 
produced by students as part of their day-to-day activities, and use this to improve 
their products, which they sell back to those systems. Data dashboards, visualiza- 
tion tools, automated decision-making systems that flag aberrant data patterns, 
and so on, require that vendors be given access to student data to test their pro- 
ducts, usually without the awareness, let alone permission, of the students 
involved. Ongoing ethical dilemmas include how public systems guarantee data 
privacy, the ethics of seeking consent from these students and their parents for 
data to be used in these ways, and who ultimately benefits from these technolo- 
gical systems that rely on data produced for “free.” 

The solution seems to be that students, teachers, or principals could come to own 
their data and decide who and how it is used. How would the system change if the 
onus was on the vendors to convince the data producers that it was in their best 
interest to provide their data because it would benefit them in tangible ways? This 
also speaks to a key aspect of technical democracy (Callon et al., 2009; Thompson 
et al., 2019), a recognition that there are significant political and ethical issues at 
work, and the OFAI represents the potential for far more complex, technical assess- 
ments, which represent the new frontier of commercial investment in education. As 
assessment systems become more and more advanced, indeed as they learn strategies 
for optimization themselves, how can the teacher, parent, or even the student make 
sense of how they work, how decisions are made, and how meaning is formed on 
the basis of adaptive algorithms? A result of datafication is that teachers are finding 
that their professional life is accelerating, partly due to the power of the displaced 
authorities. An approach grounded in the philosophy of technical democracy appears 
to be a hopeful response. 

It is important at this point to stress that we should be cautious of framing the 
emergence of new assessment regimes such as the OFAI in terms of good/bad bin- 
aries. As Foucault said, the point of critical work “is not that everything is bad, but 
that everything is dangerous, which is not exactly the same as bad. If everything is 
dangerous, then we always have something to do” (Foucault, 1983, p. 231). The 
French philosopher Bernard Stiegler has argued that the problem with the speed of 
technological change, and its proliferation and seemingly omnipresence in the 
human world, is that culture is reprogrammed by new technological systems in any 
given epoch. Stiegler’s argument is that culture is no longer keeping up with, and 
therefore, renewing, technics, causing an alienating disorientation when technology 
becomes an end in itself. “Contemporary disorientation is... linked to speed, to the 
industrialization of memory resulting from the struggle for speed, and to the specifics 
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of the technologies deployed in that struggle” (Stiegler, 2009, p. 7). How the OFAI 
lives up to its stated aim of producing a new culture based on “a suite of professional 
development options for teachers and school leaders to support and assist effective 
implementation” (ESA, 2020, n.p.) depends on how it is able to step outside the 
prevailing policy culture of assessment as and for efficiency. This move is essential to 
install an ethical system that considers rights, obligations, and ongoing commitments 
to learners, teachers, parents/carers, and the wider community. 


Conclusion 


The concern with arguing for the end of a policy or individual test is always 
about what will replace it with regards to the problems that it claims to solve. I 
have argued that the ways that next generation assessments combat ongoing 
problems of displacement and of data ownership will be critical. Without 
addressing these problems, there is a danger that next generation assessments 
will become instruments that simply recreate the problems of summative, stan- 
dardized testing that they are partly meant to solve. For example, using the 
results to create league tables, narrow curriculum, or create perverse incentives 
because individuals are being held to account for test results will likely mean 
that the OFAI fails in creating new assessment cultures. Simply doing the same 
things, faster, or at more points in time, or relying on automation that effec- 
tively “black boxes” (masks or obscures) the bases of judgement and decision- 
making in assessment is unlikely to solve the problems of displacement. In 
thinking about the OFAI, we need to consider what has been learned from 
decades of standardized testing. We also need to interrogate how likely it is that 
the next generation of assessments is going to overcome the problems that 
defined previous attempts to use data to steer education at a distance. The OFAI 
is a very interesting and dangerous development within the Australian context. 
The extent to which the platform will overcome the problem of displacement is 
a concern. The first and second order effects that this assessment ecology is 
going to have on schools and schooling still remain to be seen. 


Notes 


1 Algorithms can be designed and self-productive in relation to machine learning. 
2 It should be noted that there is no approved body of data or curriculum methodology 
related to the progressions. 
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Provocations 


Provocation #1: Digital Education in the Aftermath of COVID-19: 
Critical Hopes and Concerns 


Neil Selwyn 


Introduction 


These are difficult times to be attempting to make sense of the “digital disrup- 
tion” of education. I write this in June 2020, a moment when some people are 
beginning to entertain hopes that we have seen the worst of COVID-19. Of 
course, with hindsight it might well transpire that we are only a short way 
through the full pandemic. So far, things have been changing so rapidly that 
anything that is written runs the risk of appearing hopelessly off-the-pace by the 
time it is read. For the time being, then, it feels unwise to presume to have a clear 
critical grip on any aspect of society. As Dave Beer (2020) recently wrote: 


beyond the speed of change and a lack of focus, there is also a sense that the 
thing I’d normally be analyzing — society — will not be the same... It’s hard 
to do sociology and social science when you aren’t quite sure what the social 
is and how it is working. 


(para. 8) 


Yet regardless of how this current crisis eventually unfolds, I am fairly sure that it 
makes little sense to carry on critiquing the digitalization of education in exactly 
the same ways that we were a few months ago. Without wishing to descend into 
hyperbole, it seems sensible to assume that things are not going to be the same 
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again. Of course, all of the broad issues and concerns that were guiding our work 
prior to COVID-19 remain as important as ever. However, we need to rethink 
and reframe such matters through the lens of this pandemic and its aftermath. 
Judging by the industry gambits and political powerplays that have taken place 
around EdTech during the opening few months of the pandemic, it seems clear that 
COVID-19 is going to have a lasting impact on the ways that education and digital 
technology come together for years (if not decades) to come. Current conversations 
around remote schooling and online university teaching convey important shifts in 
tone, pace, and intent that need to be factored into any critical discussions of digital 
education. Despite the prevailing rhetoric from those who stand to gain most from 
such changes, this is not a “new normal.” Instead, we find ourselves thrust into a 
period of fast-moving reform of public education, and we need to react accordingly. 
As has been observed across most areas of life, this pandemic is bringing many 
long-simmering political tensions and social struggles to a head. So, for the fore- 
seeable future, any construction of the “digital disruption” of education needs to 
be focused firmly on ensuring that the pandemic is not used as an excuse to push 
through further corporate reforms of public education. Instead, those of us 
involved in the critical study of digital education need to work quickly to 
develop counter-narratives and alternate agendas to foster hope that this extra- 
ordinary event might perhaps act as a catalyst to reimagine better forms of public 
education for us all. Let me explicate these two points in a little more detail. 


Point 1: COVID-19 as cover for the Corporate Reform of Education 


The first few months of the pandemic saw a range of long-established technolo- 
gical trends, plans, and agendas being accelerated under the guise of being 
“emergency short-term fixes,” but nevertheless looking set to be more permanent 
in nature. Beyond education, this logic is evident in the haste with which popu- 
lation surveillance, tracking, and tracing apps were deployed hurriedly without 
due oversight and regulation. In educational terms, this logic is apparent in the 
haste with which universities have been coerced by governments to pivot to 
industry-relevant short courses, ditch unprofitable humanities and arts provision, 
merge institutions, or even close altogether. 

In terms of digital education, then, there needs to be sustained scrutiny of the 
emergency actions and logics that are being put into place. In particular, we need to 
pay close attention to how the global pivot to “emergency” forms of remote teach- 
ing during the first half of 2020 is subsequently being used as justification to radically 
rethink the future shape and character of education provision. As Frederick Hess 
(2020) observed in the initial weeks of the lockdown, “a few education analysts have 
started to sound positively giddy about this exciting opportunity to spitball ideas and 
try out nifty new programs” (para. 3). Thereafter, EdTech gurus such as Sal Kahn 
proved remarkably quick to talk of the “silver lining” of COVID-19 being a mass 
uptake of what is beginning to be termed “direct-to-student” technologies (cited in 
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Pound, 2020). Governor of the US state of New York Andrew Cuomo channeled 
this sentiment when arguing for a permanent switch-over to sophisticated “remote 
learning” across the education system. He observed: 


the old model of everybody goes and sits in a classroom and the teacher is in 

front of that classroom, and teaches that class, and you do that all across the 

city, all across the state, all these buildings, all these physical classrooms 
(Kelley, 2020, para. 6), 


then he asked, “Why, with all the technology you have?’ Regarding the potential 
of this technology, he added, “It’s hard to change the status quo. But you get 
moments in history where people say, ‘OK, I’m ready. Im ready for change. I 
get it.’ I think this is one of those moments” (para. 6). 

Such statements betray serious intent to significantly alter the conditions and 
character of public schooling through the widescale digitalization of education 
provision. Commentators in the United States are rightly horrified by the idea of 
Eric Schmidt and the Gates Foundation being given a lead in the reform of New 
York schools, yet it is understandable that state authorities and city leaders around 
the world are keen to follow similar courses of action. Switching to data-driven 
“personalized” forms of blended education provision offers a likely cost-saving 
measure in the face of an impending global financial meltdown. Yet the digital 
education infrastructures that we choose to erect in response to the current crisis 
will come to (re)define public education for decades. As Woodrow Hartzog 
(2020) argued with regards to the rapid rush to COVID-19 surveillance and 
tracking, once any technological infrastructure is established, there is an inevitable 
inertia to roll it back at a later date. “Norms get set and practices and tools 
become entrenched... industry and government [rarely] have the resolve and 
humility to double-back and try a different approach” (para. 12). 

The exact nature of the “digital solutions” that will be pushed onto public 
schooling over the ensuing months and years of the pandemic and its aftermath 
remain to be seen. However, these are unlikely to involve bespoke new forms of 
technology that have been designed carefully and sympathetically to address the 
social frailties and economic fault lines that the COVID-19 crisis has exposed and 
exacerbated. Instead, any “new solutions” are likely to rely heavily on the 
repackaging of EdTech products that have long been in the ether, such as per- 
sonalized learning systems, learning analytics, online adaptive assessments, online 
exam proctoring, etc. As such, COVID-19 is already being used as an opportu- 
nity to reanimate ideas and logics of digital education reform that had been long 
pursued prior to the pandemic by the likes of Gates, Schmidt, and those who 
follow in their wake. 

So, it follows, that policymakers and education leaders appear to be setting off 
down dangerous paths toward realizing many of the key concerns that have been 
raised within the critical studies of education and technology over the past few 
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years, not least the restrictions of proprietary platforms, classroom automation, 
surveillance-led teaching, and data-mining of lucrative student information. 
Concerns here range from straightforward profiteering through to less obvious 
musappropriations of digital technology in ways that sideline (or overlook com- 
pletely) structural inequalities. As Moeller and Tarlau (2020) reasoned: 


while reimagining and redistributing educational resources and opportunities 
is imperative, research shows that philanthropic experts often work to find 
technical solutions to systemic inequities without addressing their underlying 
causes. If we are to truly transform our nation’s inequitable educational 
system, turning to philanthropists with a track record of failing to improve 
public education is not the answer. 


(para. 8) 


If taken to their logical conclusion, these shifts will fundamentally alter the con- 
ditions and characteristics of public education, hastening the implicit agenda of 
corporate education reform that has underpinned much of the EdTech “innova- 
tion” over the past 15 years or so. Indeed, the corporate and philanthropic actors 
that are now being called upon to lead the reconfiguration of post-pandemic 
schooling have long-standing ambitions for the digitally driven “unbundling” and 
“transformation” of mass public education systems. Critical education scholars, 
therefore, need to be hypervigilant of the ways in which COVID-19 will be 
misused to force radical education reforms by those who stand to profit directly 
from them. 


Point 2: Establishing Alternate Agendas and Counter-narratives 


Of course, scholarly vigilance and critical awareness are not enough. Pushing back 
against the COVID-19-mandated digital dismantling of established schooling also 
requires resistant actions. Perhaps the most helpful role that academic researchers 
can play in supporting such actions is working to establish powerful counter- 
narratives and alternate sets of discussion points. There are a few lines of reasoning 
along which such conversation might be pursued. 

For instance, a strong case can be made that COVID-19 has already done 
much to discredit the idea of the technological fix, starkly exposing the limita- 
tions of Silicon Valley illusions of digital “innovation.” As I write, the much- 
hyped rollout of COVID-19 tracing apps are failing spectacularly in comparison 
to manual tracing techniques. Elsewhere, we have abruptly seen the limitations of 
artificial intelligence (AI) systems that find themselves lacking the appropriate 
training data to adapt to these extraordinary times. In short, the initial months of 
the pandemic have starkly illustrated that there is no quick app-based solution to 
this crisis. Instead, our hopes are pinned on the necessarily slow and methodical 
route of scientists working to develop vaccines and treatments. In short, this 


204 Lenses on COVID-19 


pandemic has already proved to be an object lesson in the limitations of digital 
technology solutionism when it comes to public health. The parallels with public 
education need to be stressed repeatedly and forcibly. 

Indeed, the COVID-19-driven pivot to temporary remote home schooling 
could be reframed as a moment when a wide range of the public are now well- 
attuned to the social limitations of educational technologies when used in situ. 
One can now talk with school leaders and policymakers about the “digital 
divide” and “homework gap” within communities, and there is general agree- 
ment (rather than the usual pre-pandemic response of denial or downplaying the 
inequalities in the assumption that “we are all online now”). Similarly, one can 
talk to parents and teachers about the sub-standard nature of platform-based 
learning, and find large numbers of people with recent first-hand experience. 

In this sense, the immediate aftermath of the emergency turn to remote 
schooling provides us with a critically engaged public, who are receptive to 
difficult conversations about EdTech. This moment needs to be seized before 
the memories of the remote schooling of 2020 fade away. Conversely, it could 
also be argued that official educational response to the pandemic saw some 
precedents being set that we might like to argue are worth fighting to repeat. 
For example, it was heartening to see governments suddenly subsidizing the cost 
of laptops and Wi-Fi connectivity for disadvantaged families otherwise lacking 
access. It was also reaffirming to see countries like Australia suddenly suspend 
the nationwide standardized NAPLAN testing of children. Actions such as these 
set a precedent for alternate future policy approaches and agendas. Rather than 
kowtowing to the corporate reform lobby, a strong case might be made for the 
continuation of these other “new” logics, that is enshrining a commitment to 
continue to invest in the social safety net and establishing an education infra- 
structure built around values of care and collective support, rather than relent- 
less assessment and measurement. 


Conclusion 


Regardless of how this pandemic unfolds, it is important that such discussions 
take place and establish new lines of argument and fresh sets of imperatives. 
Above all, it is important to diversify the voices that are called upon to set these 
agendas. As Bianca Wylie (2020) wrote in the aftermath of Toronto’s “Sidewalk 
Labs” debacle, “technology procurement is thus one of the largest democratic 
vulnerabilities that exists today” (para. 8). Above all, then, it is important that we 
work to ensure that conversations about post-pandemic education are not framed 
solely around technology issues and/or led solely by technology interests. It is 
important to push the counter-narrative that the reimagining of public education 
is not a “tech issue.” Instead, these conversations need to be pivoted firmly 
toward the heart of the matter — that is, focused on the fact that education is a 
social concern. 


Lenses on COVID-19 205 


In short, critical education scholars have a clear role to play in ensuring that the 
conversations that take place around education and digital technology are radi- 
cally different from what has been promoted to date. In this sense, we need to 
ensure these conversations are led by the interests of education and society, which 
is in stark contrast to the IT industry and philanthropic actors who are currently 
being called upon as “experts” to “develop a blueprint to reimagine education in 
the new normal” (Office of the Governor of New York State, 2020, para. 1). 
Critical education scholars have raised and explored many lines of reasoning and 
sensemaking over the past 30 years that can feed into this current moment of 
uncertainty. We should not make the mistake of watching from the sidelines. 


Note 


An earlier iteration of this provocation appeared in the self-published “zine”: 
Selwyn, N., Macgilchrist, F., & Williamson, B. (2020). Digital education after 
COVID-19. Techlash #01. Digital Education Research, Monash University, 
Melbourne, Australia. 


Provocation #2: Education without Borders, Rule without Limit 


Nick Couldry 


Introduction 


If Ivan Illich, the author of “Deschooling Society” (Illich, 1973), were brought back 
to life today, he might for a moment think his vision had come true: parents tutoring 
their children at home, and universities dispensing with the classroom to teach stu- 
dents across locations and time-zones. The means for this are fast home Internet 
access and video-conferencing platforms; the pretext is the COVID-19 pandemic 
and the decision of governments almost everywhere to impose social distancing 
measures since March 2020. But Illich was always wary of the possibility that taking 
education out of schools might usher in other forms of power and constraint. Like 
Illich, we, too, should be wary of the risks posed by this hasty rearrangement of our 
educational institutions, not just because of the well-publicized privacy and security 
limitations of platforms such as Zoom, but more importantly because dismantling 
the physical classroom, even for a few months, creates lasting opportunities for 
governing students and teachers more intensively, and substituting the profound 
inequalities of everyday living spaces for the common resource of the classroom. 
In this provocation, I will focus on the first danger — changing the regimes of 
power and rule within digital education — because it threatens to disrupt the 
educational process itself in a profound way for all students, rich and poor, and 
disrupt the role of education in wider society. 


206 Lenses on COVID-19 


The Impact of COVID-19 


There are many reasons to think that the COVID-19 pandemic and the drastic 
measures it has prompted will transform everyday life in a lasting way: the eco- 
nomic disruption, new anxieties about travel and interaction, and the possible 
dismantling of the real estate investments that have, until now, enabled the spaces 
where much of daily life outside the home goes on. But there are also reasons to 
see the COVID-19 shock not as disrupting, but as simply reinforcing trends that 
were already under way. In the lockdown, most people used social media plat- 
forms to keep threads in their life going, and almost everyone has become more 
reliant on the Internet for information, and to order home deliveries of various 
sorts; however, the increase in both trends has been under way for two decades. 

Education is no different. The lockdown has provided exceptional opportunities 
for businesses that have already invested for years in online universities, conferencing 
platforms, and other digital interfaces that “support” education. When the Governor 
of New York, Andrew Cuomo, invited Google’s former Chief Executive Officer 
Eric Schmidt to lead a commission to “reimagine” New York’s education and health 
sectors in early May 2020, the emphasis was more on continuity than drastic change. 
As Schmidt stated in a news conference with Cuomo, “my own view is that we 
need to use these moments to revisit things that are not getting enough attention and 
we have systems that need updating” (CNN Newsroom, 2020, 2:11). The systems 
of digital education were very much already being built, and Schmidt’s wider goal of 
“us[ing] technology to make things better” (2:09) is a cliché, almost an article of 
faith, for large sectors of business and government that have worked in this area for 
the last two decades. While it is far from being new, the idea that societies should 
turn to technologists “to bring that kind of visionary aspect to government and 
society” (1:01) is what digital educational providers rely on to fuel their business 
growth. However, the depth of the educational transformation invoked through the 
pandemic has not yet been appreciated in public debate, and unless citizens every- 
where are alerted to it, it will, with the benefit of COVID-19 headwinds, be instal- 
led before there is a chance to challenge it. 

What has been changing in education over the past decade is not simply the 
degree of technology use, or even the degree of different surveillance. Under way 
more broadly, is a “reengineering” of the educational process (see Frischmann & 
Selinger, 2019). To highlight the level at which change is happening, let’s borrow 
a phrase from Illich’s final book “In the vineyard of the text” (1993) in which he 
recounts deep shifts in human relations to written text that took place in the 12th 
century. Illich describes his topic as a change in “the relationship between the 
axioms of conceptual space and social reality insofar as this interrelationship is 
mediated and shaped by techniques that employ letters” (Illich, 1993, p. 4). 
Substitute the word “computers” for “letters” and we capture the revolution in 
which global educational players like Pearson and IBM are involved; it is the 
axioms of how collectively we think about education that are being reworked. 


Lenses on COVID-19 207 


As Jun Yu and I detail elsewhere (see Yu & Couldry, 2020), the growth of 
digital platforms for supplementing classrooms, tracking coursework, monitoring 
pupil progress, managing timetables, and governing school processes, is potentially 
reshaping the whole nature of education. I write this not as a specialist education 
researcher, but as a sociologist of media and data practices who is increasingly 
concerned with the rise of data colonialism as a disruptive force in society world- 
wide (see Couldry & Mejias, 2019). The claim that it is on digital platforms that 
many aspects of education are best managed — including basic interactions between 
students and teachers — is not just an innocent technological supplement to the 
classroom’s integrity, but a bid to substitute the platform for the classroom as a 
source of pedagogic knowledge. The colonizing logic of platforms knows no 
boundaries. As legal theorist Julie Cohen states, in the commercial context, “plat- 
forms do not simply enter markets, they replace (and rematerialize) them” (Cohen, 
2019, p. 42). Public sponsored domains like education will fare no better. 

The result is to disrupt the public values that, even in the world’s most marketized 
society, the United States, shape and form education (Van Dijck et al., 2018). But 
arguably, the result is even wider: to dismantle and reassemble the components of 
education as a process. Consider first the space of the classroom, until now it has been a 
space managed by the eye and voice of the teacher. Second, the resources of teaching 
have now become exclusively sourced online. Third, the time-space domain where 
evaluation takes place, has shifted from discrete evaluation periods within the broader 
teaching timetable to continuous processes of monitoring that are conducted entirely 
online and according to system constraints. Finally, the evaluative authority of the 
teacher has changed as they become a manager of data, or as Max Ventilla, the founder 
of the Alt School movement in the US, states, a “data-enabled detective” (Mead, 
2016, para. 31). In a system of knowledge production, it is data systems that have the 
largest power. It is true that digital platforms claim to give teachers an unprecedented 
level of surveillant power over pupils. As Impero, a digital education provider in the 
United Kingdom, reports, their product (Impero Education Pro) enables 


teaching staff to monitor students’ online activity from their screen in real-time. 
A thumbnail view of all student screens, in one central view, allows potential 
risk (or other instances of misconduct) to be dealt with as and when they occur. 

(Impero Solutions Limited, n.d., para. 8) 


However, authority in this surveillance system lies with the system, not the teacher. 
The current digital transformation is best summed up in Pearson’s vision of the 


? 


“digital ocean,” a metaphor for a world of education that has been completely 
datafied, from whose vantage-point the world before continuous online tracking 


of students is a digital desert. Authors of the report state, 


We can see the digital ocean of data slowly rising from our post on the edge of a 
historical era we call the “digital desert.” In the digital desert, data collection and 
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storage was expensive, limited, and isolated... [with] no systematic large-scale way 
to monitor outcomes... The absence of mobile computing devices and informa- 
tion networks in the digital desert inhibited the movement and comparison of data 
across social situations and groups... [Conversely,] the data of the digital ocean is 
not simply more data as we knew it in the pre-digital era... It is ubiquitous 
(coming from all manner of activity) and persistent, and it reflects social 
connection. 

(DiCerbo & Behrens, 2014, pp. i, 1-2, 6) 


As an example of naturalizing discourse, this is remarkable. To parse it lightly, the 
pre-digital teaching space where teachers could interact vigilantly and intensively in 
the classroom with pupils, but without the need to rely on continuous tracking of 
their actions (including their actions outside the classroom, for example on social 
media), becomes a desert, because it lacks continuous data generation. Conversely, 
the “full-screen” world of continuous tracking becomes a life-giving ocean, because 
it is full of data. In that old digital desert, where teachers have taught for centuries, it 
is implied, we would have eventually died for lack of data, whereas in the new cor- 
poratized world of continuous data extraction, it is implied, we will live. 


Conclusion 


But what exactly is it that will survive in education’s digital ocean? A regime of 
continuous “anytime-anywhere” rule, for sure, in which the historic authority of 
teachers to embody knowledge and humane care has been displaced by an all- 
knowing automated system; no doubt too, a process for extracting financially 
valuable data about pupils whose storage and control remains fundamentally 
unclear. But an educational process that can still hope to guide young people 
along a path towards creative imagination, reflexive freedom, and civic responsi- 
bility without fear? That is the human resource whose existence we are today 
wagering in return for the convenience of locking in the current crisis’s con- 
venient technological solutions. 


Provocation #3: The Electric “Shock” of the COVID-19 Crisis on 
Schooling 


Anna Hogan and Ben Williamson 


Introduction 


The effects of COVID-19 are being felt worldwide at the time of writing this provo- 
cation. Across more than 200 countries, 1.6 billion children have been affected by 
school closures (UNESCO, 2020). In attempting to facilitate the continuity of educa- 
tion, many systems have adopted remote learning. The rapid shift to online instruction 
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has been touted as “the biggest EdTech experiment in history,” and as the OECD’s 
head of education, Andreas Schleicher, has claimed, the “red tape” preventing innova- 
tion in education is now gone (Anderson, 2020). Many assume Klein’s (2007) theory of 
the “shock doctrine” or “disaster capitalism” is at work, in which the private sector has 
provided free-market solutions to solve the seemingly insurmountable problems of the 
pandemic. Indeed, EdTech players have sprung up as “emergency respondents” offer- 
ing free or heavily subsidized access to a vast array of educational products and services. 
The question is whether this emergency electronic “shock” to schooling will fade out 
or reconfigure education systems for the future. 

The research informing this provocation (see Williamson & Hogan, 2020) has 
identified that many actors and organizations view the current emergency of school 
closures not just as a short-term opportunity for supporting educators, students, and 
parents, but as a long-term opening for pursuing business interests, generating revenue, 
gathering data, and influencing policy agendas. The global education industry (Verger 
et al., 2016) has continued and accelerated its expansion during the COVID-19 
emergency, with potential long-term consequences for education systems, schools, 
educators, and students around the world. These developments remain in motion, and 
require more sustained study in order to understand their evolution over the longer 
term. In this provocation, we briefly highlight some key implications and questions 
arising from the project, and invite readers to read the full report for further details. 


New Global Policy Networks 


Commercial actors have played a significant role in maintaining education during 
school closures, but this cannot be interpreted simply as the commercialization of state 
education. Multiple organizations, actors, and governments have coalesced around 
shared aims that, in a compressed period, have become the core of a global policy 
agenda in which EdTech is a major component. The pandemic has created a catalyst for 
new forms of networked policy. These policy dynamics operate at global, national, and 
regional scales, involving a mixture of international multilateral organizations, global 
technology and education businesses, philanthropic grant funding and investor schemes, 
national and subnational government departments, public bodies and civil society 
organizations, as well as more “local” EdTech businesses and associations. The refor- 
matory aspirations of organizations with global scope and influence (e.g., UNESCO, 
the OECD, World Bank, Pearson, Gates Foundation, Microsoft, Amazon, and 
Google) have translated into 1) short-term emergency responses to the pandemic, and 
2) long-term reforms enabling education systems to recover from the pandemic in 
technologically transformed ways. 


Pandemic Prototyping 


New imaginaries of education futures circulated by commercial organizations and 
like-minded advocacy networks are proposing a radical “reimagining” and new 
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prototyping of education systems. While some schools might continue to use 
digital management platforms to organize teaching and learning activities, more 
interesting are the proposals that some hastily adopted online schools, like Britain’s 
Oak National Academy, are here to stay. They are pandemic prototypes for 
system-wide transformation. Oak National Academy, for example, has govern- 
ment backing plus the support of various education organizations, many with 
overtly reformatory aims, private sector links, and governmental connections. It 
has been proposed as a model for a future of schooling that would involve public 
and commercial partners working together. Oak National Academy is a clear 
example of the move towards privatization in public schooling, where govern- 
ments work in partnership with private, commercial, and philanthropic actors to 
deliver core public services. 


Private Infrastructures 


A key way that private sector education businesses and global technology compa- 
nies have intensified their commercial agendas in public education is through the 
provision of digital infrastructure for online teaching and learning and, importantly, 
school data storage, management, and analytics (e.g., Microsoft 365, Google G 
Suite, and Amazon Cloud). This has consolidated the market share of key private 
infrastructure providers, especially those that can not only host online learning, but 
also provide the back-end cloud services for long-term digital learning and school 
data processing. Even when schools re-open, many will have new agreements in 
place for technology companies to provide services and resources. These efforts are 
part of a larger project to reimagine the whole infrastructure underpinning school 
pedagogy, management, and curriculum provision, as market reformers and 
EdTech companies have capitalized on the crisis to push technical services ever 
further into post-pandemic public education (Cohen, 2020). 


Pandemic Profit Making 


The global education industry has advanced the idea of education as a sector for 
investment, profit making and management by private organizations. In particular, 
commercial EdTech businesses have profited from venture capital investment and 
rapidly escalating customer demand (Williamson, 2017). The commercial EdTech 
industry has made considerable inroads into “privatizing” schooling, with additional 
rapid growth of market opportunities during the pandemic. The emergency busi- 
ness plan adopted by the EdTech industry can be summarized as “support now, sell 
later,” where businesses are expanding their services now in the hope they might 
lock schools and parents into long-term subscriptions once the pandemic ends 
(Wan, 2020). However, this plan is uncertain, as while usage of free EdTech pro- 
ducts and services has grown dramatically, there has been a decrease in the use of 
paid offerings. The long-term “cost” of free access needs to be tracked into the 
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future to understand whether schools were the short-term beneficiaries of EdTech 
social responsibility, or whether the EdTech industry was able to capitalize on the 
dramatic growth of paid subscriptions. 


Conclusion 


Beyond the few implications we have highlighted, a number of unresolved 
challenges and risks remain. The rush to embed EdTech in education is likely 
accentuating issues of data protection, privacy, consent, and exploitation, parti- 
cularly given reports in the United States that existing data privacy laws were 
waived at district-level to ensure schools could use digital platforms (Lieberman, 
2020). There are also concerns about ownership of the data being collected, and 
who is controlling the curriculum. These are not new concerns, but should be 
paramount in understanding the longer-term effects of the COVID-19 pandemic 
on global schooling. In the removal of red tape, it is important to wonder: What 
has slipped through the cracks? Indeed, this returns us to Klein’s point about the 
workings of the shock doctrine. In the rush to find solutions to this crisis, and 
often uncritical acceptance of those offered by the private sector, what technol- 
ogy-based policies are being pushed through — as part of an electronic shock to 
education systems — that might deepen education inequalities and enrich the elite? 


Acknowledgement 
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Provocation #4: Teachers, the Anti-heroes? The Global Pandemic 
Crisis and the Construction of Teachers as the Problem “Other” 


Sotiria Grek 


Introduction 


At the time of writing this text, a lethal global pandemic is still unfolding, claiming 
hundreds of thousands of lives. As we struggle to achieve some kind of understanding 
of the “new normal,” we are confronted with the swift construction of novel dis- 
course that is comparing hospital admissions and excess deaths globally and in real- 
time. Language became crucial for making sense of our rapidly changing circum- 
stances: for example, public health messaging on how to take the necessary precau- 
tions against the disease transformed into a tool for unity, gave some sense of control 
over chaos, and offered hope. Many new terms, like “social distancing,” “hotspots,” 
“infection rates,” the infamous “R number,” among so many others, have become 
commonplace not only in the media but also in everyday talk. COVID-19 is not only 
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a pandemic disease; it has also ignited an epidemic of new terms and concepts 
(Treichler, 1987), the majority of which are the linguistic signifiers of quantitative 
measures. As everyday life and politics become “scientized,” this is still a world of 
uncertain public health data that, no matter how incomplete and unreliable, it is still 
the best we have and can depend on. 

However, the new pandemic discourse is not only built on the numerical 
construction of this new reality. Coupled with a renewed trust in science, it is 
also constructed on the basis of a moral discourse that glorifies the work of “key 
workers” (sometimes called essential workers). These are not only doctors and 
nurses, but also hospital cleaners, supermarket workers, bin men and women, 
delivery drivers, social care staff, and plenty more. In many countries around the 
globe, key workers were described as “heroes,” on the efforts of whom we all 
rely, in order to maintain the pillars of a functioning society’. We all stepped out 
of our front doors and onto our balconies to thank them, by clapping and pro- 
musing that we would not forget their contribution to the “nation.” Within a few 
weeks, “key workers” became the embodied and symbolic representation of the 
belief that professionalism and a solid sense of service were the cornerstones of a 
well-functioning society that could shield its citizens from harm. 


The Changing Narrative 


As the crisis has further engulfed the globe and showed its catastrophic potential, the 
war rhetoric reached a new intensity; this is a battle that we could only fight with the 
heroic efforts of those at the “frontline.” Nonetheless, teachers were perhaps the only 
group of professionals, still working either from home or in “hub schools,”” that did 
not receive the same amount of praise or attention. At least in the United Kingdom 
(UR), the case of which primarily informs this piece, there was little media mention of 
the key role of teachers in supporting parents with home schooling and looking after 
the children of key workers. Indeed, in England, when the first steps towards exiting 
the lockdown were being planned, the focus quickly turned to the need to re-open 
schools and for teachers to “return to work,” as if they were absent before. Re- 
opening schools, after all, is seen as the only possible route to restart the economy. 
After a disastrous handling of the pandemic in the UK, with the country 
mourning many tens of thousands of deaths, the teacher unions questioned the 
government’s claims that it would be safe for schools to reopen. Their doubts 
were soon backed by the governmental Scientific Advisory Group for Emergen- 
cies (SAGE; Government of the United Kingdom, n.d.), who warned about the 
dangers for reopening schools both for staff and students, but also the population 
as a whole (Toua, 2020). It is important to note that unsafe working environ- 
ments have been one of the key UK public debates and criticisms of the gov- 
ernment from the start of the pandemic; daily newspaper headlines and reports 
discuss the lack of personal protective equipment for medical staff and care 
workers. However, when teacher unions expressed their justified fears of teachers 
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getting sick, they were immediately faced with a backlash of critical commentary 
(see Whittaker, 2020), which shocked even those of us aware of the systemic 
distrust that the English education system has towards its teachers. Teachers 
became the anti-heroes, those who were “scaremongering” about the dangers of 
reopening schools, and those who were not patriotic enough to “do their duty,” 
as other workers did (Chakrabarti, 2020). Even worse, the government rhetoric, 
with Gavin Williamson, the Secretary of State for Education at its helm, went 
further and attacked teachers by almost pitching them against the general public 
in a blame-game. A letter addressed to Williamson by a teacher of a secondary 
school is telling of the depth of feeling among teachers in England. The letter 
stated, “You are our voice, yet you did not speak for us. You spoke in divisive, 
hyperbolic rhetoric of how ‘we owe it to the children’ and that the unions had a 
‘duty’ to get teachers back to school” (Collins, 2020, para. 9). The letter con- 
tinued, “It is unhelpful and unwarranted attacks on teachers like this that 
demonstrate both a lack of leadership and a lack of understanding from people 
running our country. We are still working, much like parliamentarians are: at a 
distance” (para. 9). 

None of the above would have been worrying if it were not for the specific 
national context within which they have happened; these sorts of “rows” 
between unions and the government are commonplace. Yet, there is an impor- 
tant undercurrent here that needs our attention. Teachers as professionals have 
been under continuous attack in England since the mid-1970s, when teachers 
were no longer trusted to operate with a degree of autonomy justified by their 
expertise. Conservative governments’ reforms of the ‘80s and ‘90s exacerbated the 
distrust in teachers by pushing for school improvement through the creation of 
“quasi-markets” (Furlong, 2005). New Labour’s “third way” further “de-pro- 
fessionalized” teachers (Ozga, 1995) with an emphasis on “raising standards” by 
increasing the datafication of education and its performance micromanagement. 
These developments, coupled with the rise of international comparative assess- 
ments and large international organizations’ reports on the need to improve tea- 
chers’ “effectiveness” (OECD, 2005), meant that teachers in the UK have, for 
decades, been represented in public discourse as the weak link and as untrusted 
professionals in need of reform. 


Conclusion 


The global pandemic has not only been a major crisis of public health; it has 
already had a very large impact on education, the implications of which will 
unfold for a long time to come. One of the most discussed consequences is the 
tight grip of the global education industry, which has seen the crisis as a major 
opportunity to push for new digital infrastructures of teaching and learning, 
through the development of learning management systems and online teaching 
applications. These new promises have, yet again, captured the attention of 
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policymakers and politicians who always seem to long for new education reima- 
gining, alongside, of course, their powerful non-state philanthropists and friends. 
New York Governor Andrew Cuomo “rocked the education world” (Strauss, 
2020, para. 1) by questioning why school buildings exist. He said, 


The old model of everybody goes and sits in the classroom, and the teacher 
is in front of that classroom and teaches that class, and you do that all across 
the city, all across the state, all these buildings, all these physical classrooms — 
why, with all the technology you have? 

(para. 3) 


Despite the dystopian dreams and market calculations of some education players, 
COVID-19 will almost certainly not negate the need for schools and teachers. 
However, it will lead to a major rethink of what we value and how we offer 
education, not only (or primarily) as the institution that looks after our children 
while we work, but crucially as the vital human right that education is. In a 
context of rapid change, when crises are seen as major market opportunities, we 
need to be alert to discourses that construct teachers as anti-heroes and as obsta- 
cles to learning. Schools need to open, and they will do, for they are the places 
where cheerful laughter and hope for the new post-pandemic world will be born. 
Teachers are our most precious professionals who will get us there. 


Note 


A shorter iteration of this provocation appeared in the Social Policy Association 
blog, June 4, 2020 (see Grek, 2020). 


Provocation #5: The COVID-19 Pandemic Creates Opportunities to 
Repair the Infrastructure of Public Education 


Sam Sellar 


Two well-worn lines kept coming to mind as I tried to make sense of the 
COVID-19 pandemic during the lockdown imposed across the UK in March 
2020. The first is attributed to Lenin, who apparently claimed, in relation to the 
Russian Revolution, that “there are decades when nothing happens; and there 
are weeks when decades happen.” The second was “never let a good crisis go to 
waste,” which is often attributed to Churchill but has a longer history. Both 
statements express the truth that real change is rare, and every crisis is an oppor- 
tunity. The pandemic has certainly created new opportunities for commercial 
actors to benefit from the disruption of schooling and higher education. How- 
ever, in this short provocation, I draw on Shoshanna Zuboffs (2019) analysis of 
surveillance capitalism to suggest that the crisis may also create opportunities to 
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repair the infrastructure of public education, the very infrastructure that education 
technology companies seek to disrupt. 

During the early months of the pandemic, Ben Williamson (2020) provided 
insightful analyses of how “emergency EdTech” moved quickly to create “pan- 
demic markets” and embed their digital platforms, rehearsing the familiar promise 
that “education is broken, tech can fix it” (para. 12). Many people quickly came 
to rely upon new technologies to teach, learn, and meet after the pandemic 
caused the closure of schools and universities. The acceptance and embedding of 
these technologies over a few months may indeed have resulted in change that 
could otherwise have taken decades. 

The increased use of online platforms creates more and more opportunities for the 
collection of data from us. Data is now the most coveted source of value for tech- 
nology companies, and it is the lifeblood of surveillance capitalism, which involves 
locking customers into digital platforms. The use of these platforms enables a process 
that Zuboff (2019) describes as “rendition”: “the concrete operational practices 
through which dispossession is accomplished, as human experience is claimed as raw 
material for datafication and all that follows, from manufacturing to sales” (p. 233). 
As we use digital interfaces every day, our experience is “rendered” into data, and we 
are forced to “surrender,” often unknowingly, to the logic of surveillance capitalism. 
Rendition destroys our privacy by privatizing our information. The pandemic 
should give us even more cause for concern about data privacy in education, because 
the widespread move to online teaching and learning provides an incentive for 
companies to accelerate the roll out of products and business models that are pre- 
mised upon rendering as much data as possible from users. 

Zuboff (2019) suggests three strategies for resisting the advances of surveillance 
capitalism and protecting our privacy: taming, hiding, and indignation. We can seek 
to tame the power of technology companies through legal instruments such as the 
European Union’s General Data Protection Regulation. However, it is not easy to 
tame large and powerful companies, and the capacity of regulations to effectively limit 
the exploitative practices of companies remains to be seen. We can also hide from the 
gaze of surveillance capital by disconnecting from digital platforms and masking our 
identities and activities. However, hiding implies acceptance of what we are hiding 
from and often involves the use of other platforms and software to protect us. The 
pandemic further undermines both strategies by creating a state of exception in which 
we are encouraged to set aside concerns about data privacy in order to sustain “‘busi- 
ness as usual,” which requires agreeing to a proliferating array of end-user agreements 
that enable rendition. This leaves us with indignation. Zuboff (2019) argues that an 
important condition for the emergence of indignation is the recognition that advances 
in surveillance capitalism are not inevitable. Indignation can arise as dissatisfaction with 
the mediation of our lives through digital platforms, and the rendition of our experi- 
ence into data, “teaches us how we do not want to live” (p. 524). 

The pandemic is a particularly propitious moment for such a lesson. The rapid 
extension of digital infrastructures for learning is occurring at the same time that 
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many students and families have felt the loss of physical spaces dedicated to edu- 
cation. Schools and universities are critical “public things” that provide a “world- 
stabilizing infrastructure” for action in concert (Honig, 2017, p. 96). These 
institutions serve as spaces in which we teach and learn collectively, as opposed 
to the personalization of learning that education technology companies pro- 
mote. As Susan Leigh Star (1996) observed, we generally only notice infra- 
structure when it breaks down. The shutting of schools and universities created 
a “glitch” in this critical social infrastructure that drew attention to aspects of 
education that were previously taken for granted. The question is whether we 
can repair the infrastructure in ways that do not merely restore what was taken 
for granted, nor replace it with alternatives, but instead strengthen the publicness 
of education that becomes more noticeable when spaces of public education are 
closed (Berlant, 2015). 

Every crisis is an opportunity, and almost always more than one. While the 
COVID-19 pandemic has created fertile conditions for further privatization in 
schooling, and for the expansion of business models and digital platforms pre- 
mised on extracting data from us, it might also teach us how we do not want to 
live. Those concerned with public education have an opportunity now to high- 
light the problems of commercializing education amidst a structure of feeling that 
is more acutely attuned to the possibility of decades changing in weeks, and the 
transitory nature of what so recently felt solid. Changing the way we educate 
feels more possible now that schools and universities have been forced to do so. 
Moreover, students, educators, families, and communities around the world have 
been affected by the disruption to their education caused by the pandemic, and 
many of them must surely feel dissatisfied with the digital alternatives on offer. 
The crisis thus also creates fertile conditions for recognizing that a gradual dis- 
possession of our educational experiences through the growth of digital platforms 
and personalization is not inevitable. This recognition could provide a basis for 
collective efforts to repair the infrastructure of public education in ways that 
strengthen its most important qualities. 


Notes 


1 It is interesting that in some national contexts, teachers were included in the key 
workers category. This has been the case in Australia, where the valuing of and respect 
for teachers has grown substantially during the pandemic and the necessity of schooling 
at home. This point demonstrates the significance of path dependency in the valuing or 
otherwise of teachers. 

2 During the coronavirus outbreak, hub schools in England and Scotland were the 
schools that remained open to cater for the needs of the children of key workers. 

3 Schools in England closed on Friday, 20 March 2020 and were instructed to open 
again on the 1" June 2020. Some local authorities in England, especially in the north, 
decided they would not reopen, due to a higher infection rate in these regions and 
hence the associated dangers for teachers and students alike. 
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GLOSSARY 


In producing this Glossary, we have drawn on the writing of the authors of the 
chapters in this collection, along with other publicly available definitions and 
descriptions. The latter include the OECD, various other agencies, and other 
authors (see references below). 


Algorithms Mathematical processes designed to solve a problem. They can be 
implemented computationally and configured for the purposes of achieving 
particular tasks, developing decision-making rules, and producing predictive 
and potentially actionable insights. While algorithms are initially human cre- 
ated, once they are created, and in use, they are able to connect with other 
algorithms in a self-generating process as part of machine learning and analysis 
of big data. 

Artificial intelligence (AI) No single definition. This is because every digital 
tool including AI is created for its own distinct purposes and has its own 
unique process (Leins, 2020). However, as defined by the OECD, AI is “a 
machine-based system that can, for a given set of human-defined objectives, 
make predictions, recommendations, or decisions influencing real or virtual 
environments. AI systems are designed to operate with varying levels of 
autonomy” subject to their purposes (OECD, 2019). 

Artificial Intelligence in Education (AIEd) A specialist field in AI to enable 
more personalization and flexibility, and to create more engaging learning 
opportunities with the aim to automate and eliminate the more routine tasks 
of teaching such as multiple-choice testing and scoring. 

Big data Generally refers to large volumes and variety of data, unstructured or 
structured, and too big for common software and thus usually cloud stored, 
that are produced continuously and at high velocity. 
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Bioinformatics An interdisciplinary field in which software tools and methods 
are developed for working and understanding large and complex sets of bio- 
logical data. Examples in the context of education, include eye tracking soft- 
ware, tracking of neural pathways, and software and hardware for tracking 
physical responses. 

Biometrics The automated collection of biological data related to human 
characteristics, including facial recognition, body temperature, and perspira- 
tion. As a result, it is often used as a form of identification and control. 
Examples includes body scanners and automatic temperature checking at air- 
ports, and face identification scanners for use at borders. 

Blockchain A decentralized database system that links existing or previous 
blocks of information. 

Clickstream data The list of pages viewed in order by visitors to a website from 
a series of their mouse clicks. In an education assessment context, this refers to 
data collected from how test takers and users of other online programs address 
questions, the time they take, the mistakes they make, and how often they 
access them. Thus log-file and click stream data can record online actions and 
events in real time and produce psycho-emotional data about users and students. 
The data can be used for purposes of assessing personality traits, attitudes, cog- 
nition, and abilities. 

Computer Adaptive Testing (CAT) Online tests that use branching to match 
test items with student ability based on responses to initial and subsequent 
questions. Algorithms determine a student’s pathway through a test in deter- 
mining the appropriate level of difficulty of the question sequence. 

Data An elusive term — data can come from both analogue and digital sources. Data 
can be referred to as “numerical information in digital formats” (Sellar, 2017, p. 341). 
Data become information and evidence when systems, school leaders, and teachers 
infer meaning from them, use them and take action accordingly. 

Data analytics The science of analysis of data using an algorithmic process to 
derive meaningful correlations and insights for decision-making and action. 
Data dashboard A customizable information management tool designed for 
centralization and easy visualization of data for tracking key performance 
indicators. A data dashboard provides an interactive interface for tracking, 

measuring, and extracting applicable data in an accessible way. 

Datafication Refers to the technical processes involved in the rendering of 
experience as data, which is then digitalized through software. This process 
raises philosophical and translation issues; for example, can and should all 
human experience be rendered as data (Zuboff, 2019)? 

Data infrastructures Refer to digital storage systems that enable sharing, con- 
sumption, and use of data across networks of objects and of people. Used in plural 
form to acknowledge the variety of systems in the context of interoperability. 

Data literacy Knowledge and skills to interpret, understand, and apply data so as 
to transfer into decision-making and inform action. 
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Data science When large amounts of data are curated, interpreted, and made 
meaningful for decision making purposes. 

Digital disruption The ways in which the affordances of digital technologies 
have significantly affected key aspects of the functioning of society. These 
include modes of communication, news media, the economy, industry, 
workforce, health sectors, work of governments, practices of citizenship, 
research and development in universities, and schooling. There have also been 
significant digital impacts in how individuals experience their lives, make civic 
and community contributions, experience leisure, communicate locally and 
globally and construct their identities. 

Digitization and digitalization Often seen as being synonymous but we distin- 
guish between them. We define digitization as putting online of non-digitized 
materials (e.g., taking a pen and paper test online using a USB), while we see 
digitalization as referring to the impact, mediation, and changes that occur when 
matters such as data and experience are digitalized (e.g., computer adaptive test- 
ing). Digitalization of assessment requires and enables different designs for assess- 
ment including multimodal approaches. These open up different ways of 
engaging young people, taking account of student differences and enabling 
assessment to be potentially more authentic. 

Edu-business Business corporation or privately-owned company (usually for- 
profit) with a market in educational based products (e.g., textbooks, educa- 
tional software and technology, data management, and infrastructures). 

Education technology (EdTech) Educational related businesses or corpora- 
tions that design, market, and sell educational technology products to gov- 
ernmental agencies and educational institutions (usually software and apps for 
testing, teaching, and data management tools and analysis for systems). 

Extended reality (XR) Real and virtual combined environments, and human- 
machine interactions generated by computer technology and wearables. An 
umbrella concept that brings AR, VR, and MR together. 

Immersive assessments (IA) Assessment that allows the test taker or student to 
be immersed within varying degrees of a 3D-based scenario, with the use of 
AR, VR, MR, and XR. 

International large-scale assessments (ILSAs) International large-scale assess- 
ments administered to school students and that test large samples to provide an 
international comparative perspective on student performance to inform national 
educational policy and practice. Examples include the OECD’s Program for 
International Student Assessment (PISA), The International Association for the 
Evaluation of Educational Achievement (IEA)’s Trends in International Mathe- 
matics and Science Study (TIMSS), and Progress in International Reading Lit- 
eracy Study (PIRLS). 

Internet of Things (IoT) Network of physical objects embedded with tech- 
nology such as microchips and linked via the Internet (often toys, virtual 
assistants). 
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Interoperability The capability, interactivity, and compatibility of software and 
computer systems used to integrate and link data. 

Learning analytics Tracks students through their data traces in relation to 
all types of online work, including testing and curriculum work. Just as 
CATs enable more “personalized” testing, learning analytics potentially 
enable “personalized” learning through feedback to teachers and students. 
What is called emotional learning analytics seeks to provide analyses 
about students based on their non-cognitive and affective experiences 
collected online. 

Machine learning (ML) ML and AI are prevalent in discussions of big data. ML 
drives AI. Here computers bring algorithms to big data analysis and surface 
patterns in the data. This is a “learning process” that sets computers up for 
dealing with increasing volumes of data. In this way the ML functions as the 
vehicle driving AI (see Chapter 1). 

Mixed reality (MR) Sometimes called hybrid reality, MR is the merging of real 
and virtual worlds where physical and the digital co-exist and interact in real 
time. 

Personalized learning Educational computer programs, software, and/or 
instructional approaches created to cater to the diverse learning needs, inter- 
ests, and contexts of individual learners. 

Predictive analytics Uses advanced analysis of historical data to produce real- 
time insights and predict future patterns of performance and behavior. 

Psycho-informatics Applies to computer science techniques, including psy- 
chological tracking, measurement, and analysis “of behaviors, emotions, per- 
sonality traits, attitudes, cognition and abilities. It makes use of behavioral data 
sources and analytical platforms employing techniques from data mining and 
machine learning to detect, characterize and classify behavioral patterns and 
trends” (Williamson, 2019, p. 66). 

Socio-technical education data imaginary A normative imaginary concerned 
with appropriate and educative relationships between education data, digital 
technologies, and the work of policymakers and educators (see Chapter 1). 

Virtual reality (VR) Immerses participants in completely virtual environments. 
Usually with viewing through a headset and the creation of sound and 
movement. 
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